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ABSTRACT 


Focusing  on  improving  the  evaluation  practice  in  the 
context  of  community  development  programs,  this  thesis 
presents  a  simplified  framework  which  provides  community 
development  practitioners  with  a  systematic  methodology  for 
evaluating  programs  in  their  field.  An  appraisal  of  evaluation 
literature  from  major  social  development  fields  related  to 
community  development  is  undertaken  as  well  as  a  critique  of 
evaluation  literature  directly  related  to  the  community  develop¬ 
ment  field.  Assuming  that  the  principles  and  methods  of 
social  action  research  are  most  appropriate  for  handling  and 
evaluating  the  characteristics  of  community  development 
programs,  an  approach  called  the  systems-process-context 
integrates  these  features  into  a  conceivable  framework.  This 
framework  conceptualizes  the  evaluation  process  related  to 
community  development  programs  into  five  major  evaluation 
tasks  and  provides  figures  and  tables  to  visually  portray  and 
clarify  the  theoretical  premises  of  the  approach.  Conclusions 
and  recommendations  specific  to  social  action  evaluation 
research  as  well  as  to  the  applicability  of  the  systems- 
process-context  approach  to  practical  studies  in  community 
development  are  offered.  An  application  of  the  framework  is 
illustrated  in  relation  to  an  actual  case  study.  Finally, 
an  extensive  list  of  references  is  provided  for  those  who  wish 
to  pursue  the  area  systematically. 
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CHAPTER  1 


NATURE  AND  PURPOSES  OF  EVALUATION 
IN  COMMUNITY  DEVELOPMENT 

INTRODUCTION 

Over  the  past  decade  or  so  social  programs  have 
increased  greatly  at  all  levels  of  government  in  Canada, 
resulting  in  an  increase  in  the  number  of  Canadian  develop¬ 
ment  projects  designed  to  effect  social  change  and  to 
prevent  or  eliminate  social  problems.  The  extent  to  which 
these  programs  are  succeeding  in  meeting  social  needs  can 
be  determined  by  an  evaluation  of  them.  The  long  term 
goal  of  program  evaluation  is,  thus,  to  improve  the  impact 
of  continuing  and  new  programs.  Consequently,  evaluation 
becomes  a  potentially  important  source  of  knowledge  and 
direction  for  the  policy  making  and  administration  process 
and  more  directly,  for  the  policy  makers  and  administrators 
themselves . 

In  North  America,  social  action  programs  are  in¬ 
creasingly  acquiring  a  community  development  orientation,  as 
agencies  extend  their  use  of  the  community  development  process 
in  the  planning,  implementation  and  evaluation  of  their  social 
activities  (Burton,  1978).  However,  it  is  apparent  that 
evaluation  of  the  impacts  and  effects  of  such  programs  has 
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been  a  neglected  aspect  of  the  community  development  process, 
and  of  the  study  of  the  community  development  field. 

BACKGROUND  TO  THE  STUDY 

A  change  occurred  in  the  political  and  intellectual 
environment  of  the  United  States  at  the  beginning  of  the 
1960's  resulting  in  a  shift  in  resource  allocation  from 
predominantly  physical  and  engineering  programs  towards 
socially-oriented  ones.  In  Canada,  this  appears  to  have 
occurred  a  few  years  later  at  around  the  time  of  the 
Centennial  of  Confederation.  The  result  was  an  increase  in 
the  numbers  of  government  programs  designed  to  develop  and 
enhance  the  quality  of  community  life  in  Canada;  for  example, 
the  Opportunities  for  Youth,  New  Horizons  and  the  Local 
Initiatives  Programs  .  However,  with  the  resulting  growth 
in  the  absolute  and  relative  size  of  the  government  sector 
in  national  and  local  economies  has  come  an  increasing  demand 
for  greater  accountability  in  government.  This  demand  has 
become  more  and  more  insistent  in  very  recent  years.  Specifi¬ 
cally,  demands  have  focussed  upon  the  need  for  accountability 
in  all  areas  of  social  development  and,  as  one  important  sub 
sector  of  this,  in  community  development.  Administrators 
have  faced  increasing  pressures  from  agencies  acting  as  fund¬ 
ing  sources,  from  professional  pressure  groups  and  from  clients 
to  provide  systematic  evaluations  of  their  programs  and,  in 
summary,  such  extraneous  pressures  have  contributed  significantly 
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to  a  growing  interest  in  evaluation. 

However,  despite  the  growing  demands  for  program 
evaluation,  there  has  been  a  relative  lack  of  development 
of  sound  methodologies  for  this  purpose.  It  has  been 
suggested  that  this  will  seriously  inhibit  the  growth  of 
responsible  program  planning,  policy  decisions  and  funding 
(Carter  and  Wharf,  1973).  The  most  recent  literature  points 
towards  a  growing  interest  in  improving  evaluation  methodolo¬ 
gies,  especially  in  the  areas  of  community  health  and  education 
Unfortunately,  the  numbers  of  effective  evaluation  studies 
in  the  public  sector  are  only  just  beginning  to  expand  to 
any  significant  degree  to  the  community  development  field, 
which  often  does  not  consider  evaluation  as  an  important  part 
of  the  community  development  process  (Burton,  1978).  As  a 
result,  there  remains  a  serious  absence  of  evaluation  frame¬ 
works,  incorporating  classifications,  criteria  and  indices 
applicable  to  meeting  the  needs,  objectives  and  perspectives 
of  the  community  development  process. 

ORIGINS  OF  THE  STUDY  AND  ITS  RELEVANCE 
TO  COMMUNITY  DEVELOPMENT 

The  shortcomings  of  the  state  of  evaluation  practice 
in  community  development  have  been  documented  by  Lafleur  (1977) 
Lafleur's  study  was  primarily  a  review  of  available  evaluation 
literature  related  to  community  development.  He  pointed  out 
that  the  lack  of  evaluation  methodologies  with  a  community 
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development  orientation  has  been  responsible  for  only  a  few 
effective  evaluations  being  performed  in  the  community 
development  field.  Moreover,  he  indicated  that  those  evalu¬ 
ation  studies  that  have  been  carried  out  in  community  develop¬ 
ment  were  as  low  if  not  lower  in  calibre  than  those  done  in 
other  social  fields.  Lafleur  also  noted  that  the  minimal 
amount  of  evaluation  literature  related  to  the  community 
development  field  deals  primarily  with  theoretical  concepts 
and  reflects  an  interest  mainly  in  problem  definition  rather 
than  in  ways  and  means  of  applying  evaluation  to  community 
development  programs  as  such, and, in  this  way,  Lafleur  agreed 
with  Voth  (1975)  that  evaluation  in  community  development 
could  be  generally  perceived  an  "art"  rather  than  as  a  "science". 

Lafleur  cited  one  notable  exception,  namely,  the 
attempt  by  Lovell  and  Riches  (1967  -1968)  to  develop  a  frame¬ 
work  of  characteristics,  criteria  and  indicators  for  use  in 
evaluating  the  extent  to  which  the  purposes  of  the  community 
development  agency  were  achieved.  As  a  preliminary  review, 
Lovell  and  Riches  went  beyond  the  simple  level  of  problem 
definition  and  the  discussion  of  methods  whereby  community 
workers  can  obtain  information,  such  as  how  to  design  a 
questionnaire.  Specifically,  they  were  interested  in  evalu¬ 
ating  the  changes  which  community  developers  attempt  to  effect 
in  the  community  in  both  quantitative  and  qualitative  terms. 
Indeed,  they  argued  that  qualitative  change,  while  more 
difficult  to  assess  than  quantitative  change,  is  more 
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important  to  community  development.  Lovell  and  Riches'  study 
is  especially  significant  as  it  provided  an  important  starting 
point  for  the  provision  of  more  systematic  evaluation  approaches 
in  community  development  which  would  begin  to  develop  in 
subsequent  years  through  the  application  of  social  action 
research. 

It  appears  from  the  growing  body  of  literature,  that 
within  the  past  decade  "considerable  advancement"  has  been 
made  as  a  new  generation  of  community  development  practitioners 
has  attempted  to  develop  forms  of  systematic  objective  evalu¬ 
ation  approaches  for  determining  the  legitimacy  of  community 
development  programs,  to  generally  upgrade  the  quality  of 
community  development  evaluation  reports  and,  most  importantly, 
to  demonstrate  accountability,  since,  clearly,  when  the  question 
of  funding  becomes  scarce,  it  is  allocated  only  to  those  who 
can  best  demonstrate  advantageous  results  (Reyburn,  1975). 
Furthermore,  such  systematic  methodologies  reflect  many  of  the 
attributes  of  social  action  research  whose  application  appear 
to  be  more  appropriate  for  evaluating  community  development 
programs  than  other  forms  of  traditional  research,  such  as  pure, 
basic,  or  operational  (Burton,  1978).  For  example,  in  social 
action  research,  there  is  an  emphasis  upon  the  utilization  of 
a  systems  approach  (Carter  and  Wharf ,  1973),  which  permits 
the  administrator  to  handle  conceptually  the  value  laden  nature 
of  program  evaluation  and  of  the  evaluation  process  itself. 
Burton  (1978:  45)  suggests  that  such  approaches 
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can  "beneficially  incorporate  many  of  the  same  features 
and  characteristics  found  in  the  community  development 
process  itself".  His  use  of  a  systems  approach  to  evaluation 
in  community  development  allows  him  to  differentiate  between 
criteria  of  perf ormance/outcome  and  criteria  of  value.  The 
latter  emphasize  qualitatitive  assessment,  the  importance 
of  which  to  community  development  was  recognized  earlier  by 
Lovell  and  Riches  (1968:  4). 

However,  Burton  (1978)  like  many  other  community 
development  practitioners,  is  reluctant  to  determine  more 
precise  measures  and  simply  points  out  as  did  Lafleur,  that 
such  undertakings  should  be  performed  in  order  to  provide  for 
more  systematic  evaluation  frameworks  in  community  develop¬ 
ment.  It  is  this  desire  to  develop  practical  evaluation 
frameworks  and  procedures  for  use  in  community  development 
that  has  prompted  the  present  study.  Its  aim,  in  short,  is 
to  improve  the  state  of  evaluation  practice  in  the  community 
development  field,  by  attempting  to  design  a  simplified  evalu¬ 
ation  framework  for  use  by  practitioners. 

STATEMENT  OF  THE  RESEARCH  PROBLEM 

It  has  been  suggested  by  Lafleur  (1977:  246)  that 
there  is  a  great  need  for  the  development  of  a  framework 
"with  common  classifications,  and  perhaps  criteria  and  indices" 
with  which  to  evaluate  community  development  programs.  It  is 
proposed  that  such  an  evaluation  device  should  be  based  on  a 
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systems  approach  as  advocated  by  Burton  (1978)  and  should 
give  equal  consideration  to  qualitative  and  quantitative 
measures  as  proposed  by  Lovell  and  Riches  (1967)  if  it  is  to 
assume  a  community  development  orientation.  However,  such 
measures  of  internal  accountability,  that  is,  those  related 
to  the  achievement  of  the  objectives  of  a  program,  are  not 
sufficient.  Community  development  administrators  must  take 
into  account  the  larger  environments  as  they  cannot  and  should 
not  avoid  measures  of  external  accountability  (e.g.  related 
to  the  objectives  and  goals  of  those  who  fund  programs)  upon 
which  the  financial  survival  of  their  programs  is  dependent. 
The  majority  of  community  development  programs  in  Canada  is 
funded  by  agencies  of  all  three  levels  of  government.  These 
agencies  must  be  able  to  evaluate  programs  in  a  way  which  not 
only  demonstrates  effectiveness  in  bringing  about  changes  in 
the  quality  of  people's  lives  in  a  community,  but  also  in¬ 
dicate  efficiency  in  doing  this  relative  to  alternative  uses 
of  program  resources.  This  necessitates  the  formulation  of  an 
evaluation  framework  which  considers  all  the  contextual  dimen¬ 
sions  of  the  social  system  revolving  around  and  within  a 
community  development  program  (e.g.  government,  political  and 
administrative  contexts) .  Moreover,  such  a  framework  in¬ 
tegrating  these  important  dimensions  within  its  design  can 
aid  the  practitioner  in  more  realistically  evaluating  the 
success  of  changes  effected  in  the  individual  and  community 
and  should  do  so  both  in  quantitative  and  qualitative  terms 
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through  applying  the  techniques  of  social  action  research. 

RESEARCH  PROCEDURES 

In  the  process  of  knowledge  building  for  community 
development,  one  of  the  strengths  of  the  field  lies  in  its 
interdisciplinary  and  multidisciplinary  nature:  the  opportunity 
to  choose  from  other  social  science  fields  and  social  or 
human  service  fields  the  knowledge  applicable  to  community 
development  evaluations.  The  strength  of  this  orientation 
is  the  potential  flexibility  such  choice  gives  to  the  field 
to  modify  and  apply  other  evaluation  methodologies  within  a 
framework  designed  specifically  in  our  work  with  community 
development  programs. 

The  development  of  the  problem  statement  for  the 
present  project  requires  the  conduct  of  the  following 
research  tasks.  First,  a  critical  appraisal  of  evaluation 
literature  from  the  major  social  development  fields  such  as 
extension,  social  work,  health  and  education  will  be  performed, 
in  order  to  identify  and  define  common  variables  relative  to 
the  nature  of  evaluation,  its  purposes,  related  concepts  (e.g. 
evaluation  research) and  especially  its  application  through 
frameworks  related  to  the  four  levels  at  which  evaluation  of 
social  action  programs  are  usually  undertaken:  namely, input , 
process,  product  and  context  (Longest,  1975).  Second,  an 
appraisal  of  evaluation  literature  in  community  development 
will  be  undertaken  in  order  to  develop  a  statement  and 
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definitions  related  to  the  community  development  process  and 
to  the  role  of  evaluation  in  this.  Third,  an  attempt  will 
be  made  to  synthesize  the  material  covered  in  the  first  two 
tasks  in  order  to  clarify  the  basic  premises  constituting  the 
framework  developed  within  the  following  chapter.  Fourth, 
an  outline  of  an  analytical  framework  based  on  the  variables 
identified  as  significant  from  the  first  three  tasks  will  be 
identified.  This  framework  will  be  developed  at  the  following 
levels:  first,  the  identification  of  major  evaluation  components 
second,  the  identification  of  both  quantitative  and  qualitative 
dimensions  necessary  for  the  evaluation  of  community  develop¬ 
ment  programs  based  on  task  and  process  objectives;  third, 
the  identification  of  appropriate  elements  within  the  establi¬ 
shed  evaluation  dimensions;  and  fourth,  the  identification  of 
quantitative  and  qualitative  measures  of  these  factors  (such 
as  use  made  of  existing  services  or  responsible  participation 
by  group  members) .  Conclusions  and/or  a  commentary  on  the 
potential  value  of  the  analytical  framework  for  use  in  the 
evaluation  of  the  impact  of  community  development  programs 
will  constitute  the  final  element  in  the  study. 

DEFINITION  OF  TERMS 

Since  the  fields  of  community  development  and  evaluation 
are  both  ones  in  which  some  disagreement  exists  regarding  the 
nature  and  focus  of  the  field  and  the  precise  meanings  of 
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central  terms,  it  is  essential  that  this  study  begin  by 
defining  key  concepts  as  they  will  be  employed  throughout 
the  study.  The  specific  concern  is  with  two  broad  concepts: 
first,  the  concept  of  community  development,  with  its  related 
programs  and  relationships  to  the  notion  of  "innovative" 
administration;  and, second,  the  concept  of  evaluation  and  the 
related  concern  of  evaluation  research,  as  well  as  its  relation¬ 
ship  to  social  action  research  and  community  development.  Each 
of  these  two  broad  concepts  is  employed  in  an  activity  which 
consumes  the  energies  of  a  community  development  practitioner 
associated  with  a  government  or  quasi-government  agency  whose 
responsibility  is  to  plan,  implement  and  more  importantly, 
evaluate  community  development  programs. 
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A  Definition  of  Community 
Development 

There  are  almost  two  hundred  available  definitions 
for  community  development!  This  is  so  because,  according 
to  Aldred  (1977:  241)  "the  conceptions  of  community  develop¬ 
ment  are  as  varied  today  as  the  circumstances  and  situations 
in  which  they  are  applied".  In  other  words,  it  is  argued  that 
what  constitutes  community  development  is,  in  effect,  case 
specific.  However,  there  are  two  points  of  consensus  upon 
which  all  conceptualizations  appear  to  agree;  first,  that  it 
is  a  process  of  purposive  social  change  directed  at  the 
community  level;  and  second,  that  is  differentiated  from  other 
theories  of  change  processes  in  that  it  is  assumes  that 
community  citizens  should  participate  in  the  decision-making 
processes  of  defining  and  meeting  their  own  felt  needs. 

Community  Development  Programs 

Those  activities  which  promote,  sustain  and  support 
community  action  through  collaborative  efforts  between  the 
citizens  themselves  and  the  community  development  practitioner 
are  facilitated  in  Canada  through  the  implementation  of  programs 
designed  to  create  both  quantitative  and  qualitative  change. 

Such  a  definition  of  community  development  programs  utilized 
in  this  thesis  is  supported  by  Hynam  (1972:  16)  whereby  .  .  . 

as  a  philosophy  and  method,  community  development 
offers  a  way  of  involving  people  more  fully  in  the 
life  of  their  communities.  It  generates  scope  and 
initiatives  which  enable  people  to  participate 
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creatively  in  the  economic,  social  and  cultural  life 
of  a  nation.  It  provides  above  all  a  basis  for  a 
more  profound  understanding  and  more  effective  use 
of  the  democratic  process.  These  are  essential 
elements  of  Canada's  social  policy.  These  principles 
underlie  our  current  social  programmes  which  in  essence 
are  designed  to  make  it  possible  for  people 
to  overcome  limitations  in  their  environment. 

Such  programs  facilitate  interventions  aimed  at  groups  or 
institutions  whose  goals  are  intended  to  meet  the  needs  of 
people  and  develop  human  resources  within  the  existing 
economic  and  social  system.  Here  the  stress  is  on  what 
Dunham  (1970)  has  terms  "process  goals"  rather  than  "task 
oriented  goals"  or  concrete  objectives.  In  other  words,  their 
main  intention  is  broad  scale  social  change  at  the  community 
level  rather  than  instrumental  intervention  aimed  at  the  indi¬ 
vidual  as,  for  example,  in  clinical  social  work.  These  pro¬ 
grams  can  become  highly  formalized  and  often  relate  to  such 

areas  as  health  and  welfare  (e.g.  multi-service  centres), 
agriculture,  industry,  urban  development  (e.g.  citizen  partici¬ 
pation)  and  recreation.  Also,  community  development  programs 
can  relate  to  a  number  of  informally  organized  services  (e.g. 
voluntary  associations) .  It  should  be  also  pointed  out  that 
community  development  programs  reflect  an  innovative  nature, 
an  emphasis  on  trying  out  and  learning  the  nature  of  such 
programs  which  makes  them  open  to  careful  scrutiny  for  possible 
improvements  and  thus,  there  is  more  opportunity  and  demand  for 
evaluation  (Carter  and  Wharf,  1973). 

The  effective  administration  of  programs  becomes  the 
responsibility  of  the  community  development  practitioner 


13 


associated  with  a  government  or  quasi-government  agency.  The 
problem  for  the  practitioner  then  becomes  hew  he  can  reconcile  any 
possible  arising  conflicts  within  established  policies  without  losing 
sight  of  his  beliefs  in  the  theoretical  premises  of  the  community 
development  process  whereby  people  participate  in  the  decision¬ 
making  process  of  defining  and  meeting  their  own  needs.  The 
answer  lies  in  what  I  have  termed  an  "innovative"  administrative 
approach  to  be  adopted  by  the  community  development  practitioner. 
This  concept  necessitates  involvement  and  input  from  the 
people  and,  in  this  way,  the  administrator  can  work  with  people 
rather  than  on  or  for  people  (Gibson,  1977:  31),  this  being  the 
prime  responsibility  of  a  community  development  practitioner. 

The  advantages  of  utilizing  such  an  orientation  have  been 
well  documented  by  Frederickson  (1975)  who  equates  the  principle 
of  the  community  development  process  with  those  of  modern 
administration.  Frederickson  contends  that  traditional 
development  administration  has  in  the  past  been  overly  central¬ 
ized,  authoritative  and  technological,  but  that  it  is  making  a 
transition  in  the  direction  of  the  principles  inherent  within 
the  community  development  process  which  recognize  the  need 
for  organizations  to  be  structured  in  a  way  so  as  to  remain 
responsible  to  people's  needs.  Thus,  through  an  "innovative" 
administrative  approach,  practitioners  responsible  for  pro¬ 
gram  evaluations  should  always  work  within  a  framework  which 
facilitates  input  from  the  people  rather  than  relying  on  an 
overwhelming  amount  of  technology. 
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A  Definition  of  Evaluation 

There  appear  to  be  as  many  definitions  available  for 
evaluation  as  there  are  for  community  development.  Confusion 
appears  to  arise  most  often  around  methodological  concerns 
and  an  acceptance  of  what  is  "scientific"  and  what  is  not. 

For  example,  Suchman  (1973)  defines  "scientific"  as  implying 
the  use  of  experimental  and  control  groups,  whereas  to  Hayes 
(1966)  the  term  simply  implies  objective,  comorehensive  and 
systematic  consideration  of  those  factors  related  to  the  acti¬ 
vities  of  a  programme.  In  yet  another  approach,  Sherwood 
(1967)  extends  the  meaning  of  scientific  evaluation  to  encompass 
those  skills  and  strategies  beyond  social  research.  These 
differing  definitions  simply  serve  to  illustrate  that  evaluation 
is  not  a  unitary  concept.  Indeed,  it  is  argued  that  the  term 
should  maintain  flexibility  so  as  to  meet  varied  program 
needs  and  conditions.  However,  amongst  the  discrete  definitions, 
most  conclude  that  an  application  of  judgment  to  programme 
outcomes  plays  the  primary  role  in  the  evaluation  process. 
According  to  Steele  (1975:  14)  evaluation  is  .  .  . 

a  process  of  forming  judgements  about  programs 
usina  criteria  of  standards  and  descriptions  of 
wnat  occurred  and  resulted  in  the  program. 

More  specifically,  according  to  Lafleur  (1977:  20).  .  . 

community  development  evaluation  is  the  act  and 
reporting  of  judgements  of  the  merits  of  the 
techniques  and  activities. 
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Thus ,  the  emphasis  upon  the  act  and  reporting  of  judgments 
related  to  community  development  programs  based  on  criteria 
and  standards  will  be  the  working  definition  of  evaluation 
used  throughout  this  thesis. 

The  Purposes  of  Evaluation 

A  review  of  the  literature  suggests  that  the  purposes 
of  evaluation  may  serve  one  or  more  of  the  following  functions 
as  stated  by  Suchman  (1973:  8); 

1.  To  clarify  and  re-examine  program  objectives  and 
underlying  assumptions. 

2.  To  examine  more  thoroughly  the  operation  and 
content  of  a  program  so  as  to  clarify  its  essential  parts. 

3.  To  re-evaluate  the  theoretical  program  base 
particularly  the  ways  in  which  principles  are  turned  into 
practices . 

4.  To  develop  new  hypotheses  for  future  research. 

5.  To  develop  a  more  critical  attitude  on  the  part 
of  program  personnel  including  the  chance  to  make  suggestions 
for  improving  the  program. 

6.  To  increase  staff  morale  and  committment  as  a 
result  of  their  efforts  to  improve  programs. 

The  purposes  of  evaluation  may  be  undermined  for  one  or 
more  of  the  following  reasons  outlined  by  Weiss  (1973:  10-11): 

1.  The  program  content  is  not  questionable. 

2.  The  program  has  no  apparent  clear  orientation. 
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3.  Large  discrepancies  exist  in  program  goals. 

4.  Evaluation  is  too  costly. 

5.  Illegitimate  reasons  exist  for  undertaking  an 
evaluation  such  as  trying  to  postpone  decision-making,  to 
avoid  responsibility,  to  enhance  the  prestige  of  an  agency, 
to  fill  grant  requirements,  to  undermine  program  director 
efforts,  to  attempt  "eye  washing"  (e.g.  evaluating  only  the 
good  parts) ,  or  to  "submarine"  or  destroy  the  objectivity  of 
a  program, especially  for  political  reasons. 

Pref ormative ,  Formative  and 
Summative  Evaluation 

Further  distinctions  must  be  made  at  this  point  with 
regard  to  theories  of  evaluation.  As  a  generic  term, evaluative 
decisions  can  be  made  prior  to  the  implementation  of  a  program. 
This  is  referred  to  as  the  pre-f ormative  evaluation  stage  (Alkin 
and  Fitz-Gibbon,  1975:  4,  Figure  ])  and  includes  two  major  activities; 
program  planning  evaluations  and  needs  assessment. 

After  a  program  is  implemented  it  requires  monitoring 
and  modifying  in  order  to  make  it  work  as  effectively  as 
possible.  Activities  here  focus  on  implementation  evaluation 
and  progress  evaluation  which  constitute  the  formative  evaluation 
stage  (Alkin  and  Fitz-Gibbon,  1975). 

Once  a  program  has  undergone  several  cycles  of  formative 
evaluation  it  stabilizes  and  is  adopted  as  an  ongoing  program. 

At  this  stage,  summative  evaluation,  an  assessment  of  a  program's 


success,  is  undertaken  through  two  separate  evaluation  activities 
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PROGRAM  STAGES  EVALUATION  STAGES 


Figure  1  The  Relationships  Between  Evaluation  Activities 

and  the  Stages  of  A  Developing  Program  (From  Alkin 
Fitz-Gibbon,  1975) 
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known  as  documentation  and  outcome  evaluation  (Alkin  and 
Fitz-Gibbon,  1975)  . 

Evaluation  and  Evaluation 
Research 

If  evaluation  is  then  primarily  the  making  and  report¬ 
ing  of  judgements  related  to  the  success  or  failure  of  a  pro¬ 
gram,  then  evaluation  research  can  be  simply  defined  as 
research  relating  to  evaluation.  However,  this  is  clearly 
too  broad  and  general  a  definition.  Something  more  specific 
is  required.  Many  authors  stress  that  the  difference  between 
evaluation  and  evaluation  research  lies  in  the  fact  that  the 
latter  applies  systematic  techniques,  criteria  and  measures 
to  acquire  objectivity  in  the  process  of  judging  programme 
outcomes.  According  to  Cherns  (1969)  evaluation  research 
covers  a  variety  of  techniques  and  models  which  he  has 
categorized  as  pure  research,  basic  research,  operational 
research  and  social  action  research.  According  to  Carter  and 
Wharf  (1973:  31)  social  action  research  is  characterized  by 
the  following  attributes;  it  emphasizes  the  introduction  and 
observation  of  planned  change;  it  is  problem  centered,  emphasizing 
some  improvement  or  some  socially  desirable  goal;  it  makes 
partial  use  of  empirical  science;  it  emphasizes  a  systems 
approach;  it  is  most  effective  when  applied  in  complex,  rapidly 
changing  systems;  it  is  concerned  with  variables  which  appear  to  make 
some  difference  in  the  performance  of  the  system  (e.g.  utilization 
of  human  resources,  control,  and  leadership  collaboration  and 
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conflict) ;  it  focuses  on  the  quality  and  nature  of  relation¬ 
ships  with  the  client  group  or  organization  which  is  the  target 
of  the  planned  change;  and  the  relationship  between  the  action 
researcher  and  evaluation  presents  unique  role  requirements. 

As  it  is  the  attributes  of  action  research  which  appear  to  have 
been  primarily  applied  in  carrying  out  evaluation  studies 
related  to  community  development  programs,  it  will  be  the  type 
of  evaluation  research  referred  to  in  this  thesis. 


CHAPTER  2 


A  CRITICAL  APPRAISAL  OF  EVALUATION  LITERATURE  FROM 
SOCIAL  DEVELOPMENT  FIELDS  RELATED 
TO  COMMUNITY  DEVELOPMENT 

Based  on  the  assumption  that  community  development 
is  a  sub  sector  of  social  development  and  its  activities 
reflect  those  of  a  social  action  variety,  then  there  is 
value  in  examining  the  ways  in  which  other  related  social 
development  fields  (e.g.  extension,  education,  social  work, 
community  health)  have  applied  the  evaluation  methodologies 
of  social  action  research. 

The  material  in  this  chapter  is  organized  to  cover 
what  appears  from  the  literature  review  to  be  the  four  levels 
at  which  evaluation  of  social  action  programs  can  and  should 
be  undertaken:  namely,  input  evaluation,  process  evaluation, 
product  evaluation  and  context  evaluation. 

Specifically,  as  borrowed  from  Longest  (1975:  51-53), 
inout  evaluation  refers  to  the  resource  recruirenents  of  action 
programs  as  related  to  the  setting  of  objectives.  Process 
evaluation  refers  to  examining  the  program's  operation, 
process  and  procedures.  As  it  is  the  most  demanding,  time 
consuming  and  difficult  evaluation  level,  it  will  receive  the 
greatest  attention  in  this  appraisal  in  order  to  reveal  its 
complex  elements  such  as  the  specification  of  measures,  and 


20 


21 


choice  of  study  and  research  designs  that  serve  as  an  important 
intervening  variable  between  input  and  product  levels  in  order 
for  a  complete  evaluation  to  be  undertaken.  Product  evalua¬ 
tion  refers  to  how  well  the  objectives  of  a  program  have  been 
met.  Finally,  as  Longest  concludes, in  evaluation  research 
the  context  level  provides  important  information  for  completing 
the  research  design.  It  assumes  that  the  hypothesis,  strategies 
and  likely  outcomes  from  within  an  evaluation  study  are 
dependent  upon  particular  contextual  elements  (e.g.  the  results 
of  an  action  program  may  differ  in  a  social  system  with 
multipower  structures  as  compared  to  one  controlled  by  an 
elite  group).  Hence,  the  nature  of  contextual  dimensions  helps 
to  provide  for  a  more  objective  assessment  for  the  input,  process 
and  product  levels  of  evaluation. 

These  levels  are  interdependent  and  it  appears  that 
they  consider  the  following  major  key  areas  of  decision-making 
faced  by  the  evaluator  which  must  be  examined  and  interrelated 
into  a  framework  for  undertaking  a  comprehensive  evaluation: 
first,  the  setting  of  overall  program  objectives  in  relation¬ 
ship  to  the  theoretical  premises  and  financial  concerns  of  the 
program;  second,  the  conceptualization  and  measurement  of  pro¬ 
gram  objectives  as  related  to  needs  assessment  and  program 
planning  evaluation  (pref ormative  evaluative  stage) ;  third,  an  examin¬ 
ation  of  the  implementation  and  progress  of  a  program  towards 
meeting  its  stated  obj ectives  (formative  evaluation  stage)  which 
examination  of  process  has  implications  for  the  choice  of 
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overriding  study  design  (e.g.  systems  or  goal  based  approach), 
the  selection  of  research  design  (e.g.  experimental  design  or 
descriptive,  dynamic  causal  models)  and  the  selection  of  evalu¬ 
ation  components  such  as  qualitative  and  quantitative  measures; 
fourth,  an  examination  of  the  activities  of  documentation  and 
outcome  and  their  relationship  towards  assessing  the  end 
product  of  a  program  (summative  evaluation  stage) ;  fifth,  in 
light  of  the  importance  of  context,  will  be  examined  those 
important  organizational  and  socio-political  factors  within 
and  revolving  around  a  social  action  program. 

It  then  follows  that  an  examination  of  these  four 
evaluations  levels  —  input,  process,  product,  context--as  well  as 
their  corresponding  activities  within  the  pref ormative ,  formative 
and  summative  evaluation  stages  will  be  the  primary  objective 
for  this  appraisal  and  from  this  examination  will  emerge  a 
comprehensive  review  of  the  evaluation  field,  both  in  theory 
and  practice  to  ascertain  not  only  its  present  state  but  its 
potential  application  to  the  field  of  community  development. 

This  will  assist  community  development  practitioners  towards 
examining  related  factors  between  their  field  and  other  social 
development  areas  and  will  provide  them  with  guidelines  for 
developing  a  framework  specifically  designed  to  undertake  a 
comprehensive  evaluation  of  community  development  programs 
through  applied  social  action  research. 
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INPUT  EVALUATION  LEVEL 
The  Setting  of  Objectives 

The  conceptualization  and  measurement  of  a  program's 
overall  objectives  is  the  first  requirement  in  evaluation 
research  (Carter  and  Wharf,  1973).  This  is  often  a  difficult 
undertaking  with  social  action  programs,  as  often,  according 
to  Carter  and  Wharf  (1973:  35),  such  programs  often  contain  more 
than  one  objective,  or  statements  of  objectives  are  absent;  are 
too  generally  stated;  are  unrealistic  or  unachievable;  or  are 
partial  or  inaccurate. 

The  problematic  task  of  setting  objectives  can  be 
alleviated  when  it  is  done  in  accordance  with  the  theoretical 
premises  of  the  program  which  can  act  as  an  important  guide¬ 
line  in  what  to  measure  and  how  to  measure  it  (Carter  and 
Wharf,  1973:  36).  For  example,  in  order  to  eliminate  the 
confusion  in  measuring  multiple  objectives  which  is  character- 
istic  of  most  social  action  programs  a  useful  procedure  is  to 
order  them  into  three  different  categories:  immediate,  inter¬ 
mediate  and  ultimate  goals  as  proposed  by  Suchman  (1957:  51).  Each 
category  of  objectives  has  its  own  set  of  criteria  measured  in 
terms  of  as  one  moves  from  a  higher  to  a  lower  objective  each 
lower  category  must  assume  all  assumptions  made  for  objectives 
above  it.  Each  of  the  three  categories  contains  a  variety  of 
criteria  to  which  the  failure  or  success  of  a  program  can  be 
attributed  based  on  effort,  adequacy  of  performance,  efficiency 
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and  process. 

The  product  of  the  preceeding  analysis  on  objective 
setting  can  be  usefully  structured  in  some  sort  of  hierarchi¬ 
cal  scheme  which  ranks  several  cognition  levels  of  primary  and 
supporting  activities.  One  of  the  best  examples  from  the 
existing  literature  of  such  a  product  can  be  found  in  Claude 
Bennett  s  (1975:  7-12)  hierarchy  of  evidence  for  program 
evaluation  in  extension  programs. 

Bennett  establishes  seven  categories  of  criteria  based 
on  a  chain  of  events"  for  evaluating  the  objectives  of  exten¬ 
sion  programs  extracted  from  their  thoeretical  premises. 
Similar  to  all  action  programs,  extension  programs  usually 
have  many  objectives  occurring  within  several  or  all  levels  of 
the  hierarchy  illustrated  in  Figure  2.  Through  the  following 
scheme,  Bennett  offers  guidelines  in  choosing  evidence  within 
these  categories.  First  in  the  chain  are  inputs  (resource 
expended  by  Extension) .  Activities  are  produced  from  these 
inputs  which  involve  people .  People  have  both  negative  and 
positive  reactions  to  the  program  which  initiate  changes  in 
their  knowledge,  attitudes,  skills  and  aspirations  (KASA) . 

When  people  apply  KASA  changes  to  their  living  and  working 
environment  a  change  in  practices  can  occur  from  which  are 
elicited  end  results.  These  end  results  should  encompass 
achieving  the  ultimate  aims  of  the  Extension  program.  Within 
each  category  Bennett  provides  examples  of  criteria  related  to 
the  objectives  of  that  particular  level  of  the  hierarchy. 
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7.  End  Results 


6.  Practice  change 


5 .  KASA  change 


4* Reactions 


3.  People  involvement 


2.  Activities 


1.  Inputs 


Figure  2 . 


A  hierarchy  of  evidence  for  extension 
program  evaluations  (From  Bennett,  1975) 
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Bennett  contends  that  people  value  a.  program  moire  if  nearly 
all  program  objectives  or  plans  are  met.  Moreover,  a  more 
valued  program  is  more  likely  to  attract  additional  funding 
or  "its  objectives  are  likely  to  be  retained,  intensified  or 
broadened".  Bennett  offers  the  following  guidelines  which  are 
helpful  in  deciding  how  to  evaluate  a  program:  first,  the 
evidence  of  program  impact  becomes  stronger  as  the  hierarchv 
is  ascended;  second,  evaluations  are  strengthened  by  assessing 
Extension  levels;  third,  it  is  more  difficult  and  costly  to 
obtain  higher  evidence  and  'harder'  evidence. 

In  short,  it  appears  that  the  setting  of  objectives  can 
be  performed  more  easily  if  it  is  done  in  accordance  with  the 
theory  extracted  from  the  overall  program  plan  that  provides 
guidelines  on  what  to  measure  and  how  to  measure  it. 

Also,  in  addition  to  theory,  the  priorizing  of  "what 
gets  evaluated"  should  tie  in  with  financial  considerations 
(Carter  and  Wharf,  1973).  This  important  aspect  is  "the  extent 
to  which  the  evaluator  depends  on  the  financial  commitment  of 
the  organization  and/or  the  administrator  to  a  thorough  evalu¬ 
ation".  Discrepancies  can  arise  between  the  objectives  of  the 
evaluation  and  the  project  that  can  lead  to  confusion  surround¬ 
ing  the  financial  feasibility  of  carrying  out  an  evaluation. 

For  example,  often  not  enough  funds  are  set  aside  for  meeting 
the  costs  involved  in  conducting  a  feasible  evaluation  or  often 
many  view  evaluation  research  as  too  costly  or  of  little 
practical  value  and  would  like  to  see  the  dollars  spent  on  the 
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social  problem  itself  rather  than  any  research  associated  with 
it.  In  order  to  prevent  these  obstacles,  a  consideration  of 
program  costs  can  be  usefully  determined  relative  to  achieve¬ 
ments  based  on  consideration  of  the  following  factors  listed 
by  Carter  and  Wharf  (1973:  68-69): 

1.  Time  required  for  implementation  and  effective 
operation  < 

2.  Requirements  for  additional  skilled  manpower  as 
compared  to  availability. 

3.  Input  upon  present  manpower  in  terms  of  additional 
training,  changed  workload  and  restructured  roles. 

4.  Costs  of  capital  outlays  for  facilities. 

5.  Remodelling  or  obsolescence  of  present  equipment 
and  facilities. 

6.  Operations  and  maintenance  costs  after  initial 
investment  has  been  made. 

The  input  evaluation  level  incorporates  two  other  major 
activities  which  help  to  more  systematically  conceptualize  and 
measure  program  objectives.  These  two  activities  are  needs 
assessment  and  program  planning  evaluation  (defined  as  pre- 
formative  evaluation  activities) ,  the  former  of  which  provides 
an  important  informational  base  or  input  into  the  conceptual¬ 
ization  of  objectives  and  the  latter  of  which  helps  to  intensify 
the  subsequent  evaluation  activities  to  measure  objectives  in 
accordance  with  the  way  in  which  a  program  is  planned.  It  is 
toward  these  pre-f ormative  evaluation  activities  that  this 
thesis  will  now  direct  its  attention. 
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Needs  Assessment 


According  to  Alkin  and  Fitz-Gibbon  (1975:  3)  "in  needs 
assessment,  the  evaluator  provides  information  on  the  perceived 
importance  of  relevant  goal  areas,  their  current  status  and  the 
relative  priorities  of  each". 

Siegel,  Attkisson  and  Carson  (1978:  229)  state  that 
needs  assessment  is  conceptually  and  operationally  different 
from  evaluation  in  the  fact  that  it  is  an  environmental  monitor¬ 
ing  system.  They  describe  the  differences  between  program 
evaluation  and  needs  assessment  in  the  following  way.  Accord¬ 
ing  to  Siegel  et  al .  (1978:  224),  program  evaluation  is: 

1.  The  process  of  making  reasonable  judgements 
about  program  effort,  effectiveness,  efficiency 
and  adequacy; 

2.  Based  on  systematic  data  collection  and  analysis; 

3.  Designed  for  use  in  program  management,  external 
accountability  and  future  planning; 

4.  With  special  focus  on  accessibility,  acceptability, 
awareness,  availability,  comprehensiveness, 
continuity,  integration  and  cost  of  services. 

In  contrast,  they  define  need  assessment  as: 

An  environmental  monitoring  system  that  is  .  .  . 

Designed  to  measure  and  make  judgements  about 
program  relevance,  adequacy  and  appropriateness. 

Based  on  systematic  collection  and  analysis  of 
information . 

Regarding  the  needs  for  health  and  human  services. 

As  filtered  through  multiple  levels  of  societal 
perspectives . 

As  generated  through  multiple  measurement 
approaches . 

However,  despite  their  conceptual  and  operational  differences, 
it  is  important  to  note  that  program  evaluation  and  needs 
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assessment  are  related  processes  each  giving  an  information 
base  that  is  critical  for  the  effective  assessment  of  social 
action  programs . (Attkisson ,  1978:  224). 

There  exists  a  variety  of  approaches  for  conducting 
needs  assessments  which  have  been  neatly  outlined  by  Bell, 
Wharheit  and  Schwab  (1978:  70)  as:  the  key  informant  approach, 
the  rates  under  treatment  approach;  the  social  indicators 
approach;  the  group  approaches  (e.g.  the  community  forum);  and 
the  social  area  or  field  survey  approach  (e.g.  citizen  survey) . 
The  next  section  will  review  the  latter  two  approaches  which 
emphasize  how  to  conduct  community  needs  assessments  as  they, 
unlike  the  first  three  approaches,  focus  on  eliciting  infor¬ 
mation  from  the  entire  community  rather  than  a  select  group  of 
individuals.  In  this  way,  the  action  researcher  interested  in 
community  development  will  be  familiarized  with  those  approaches 
that  he  will  most  often  use  when  conducting  a  community  needs 
assessment  (Mitchell,  1977)  in  order  to  receive  an  informational 
base  or  input  that  is  important  to  assess  effectively  the 
objectives  of  community  development  programs. 

The  citizen  survey,  which  is  one  type  of  social  area 
survey,  is  a  technique  which  accommodates  broader  citizen  and 
community  participation  in  determining  service  priorities  and 
identification  of  needs.  This  type  of  information  can  be 
yielded  from  anonymous  through-the-mail ,  stratified  random 
sampling  or  direct  interview-based  methods.  The  advantage 
of  this  approach  is  that  it  provides  scientifically  valid  and 
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reliable  information  from  individual  community  members  regarding 
specific  information  about  their  own  service  needs  and  utili¬ 
zation  patterns.  It  is  also  the  most  flexible  of  assessment 
techniques  and  the  most  direct  method  of  elicting  relative  and 
quantitative  estimates  of  needs.  On  the  other  hand,  it 
often  demands  considerable  resources,  expertise  and  organiza¬ 
tional  support  from  the  service  delivery  system,  respondents 
are  often  reluctant  to  answer  questions,  and  the  validity  and 
reliability  of  questionnaires  and  measuring  instruments  are 
often  questionable. 

Beyond  the  survey  approach,  community  views  can  be 
elicited  through  a  group  approach,  such  as  the  community  forum 
which 'according  to  Coursey  (1977:  49)  is  an  approach  that 
relies  on  the  perception  of  any  person  living  or  working  in  a 
community  whose  input  provides  a  reliable  resource  of  the 
sociological  and  psycholgical  aspects  of  that  community.  People 
are  asked  to  appraise  the  need  and  service  patterns  of  those 
living  in  the  community.  Forum  studies  are  designed  around  a 
series  of  public  meetings  to  elicit  views  from  as  many  people 
as  possible  on  a  single  issue.  The  meetings  are  structured  to 
encourage  spontaneity  and  candor  on  the  part  of  the  partici¬ 
pants.  However,  some  sort  of  guidance  is  created  by  the 
organizers  who,  prior  to  the  forum,  outline  the  specific  objectives 
of  the  program  and  prepare  a  set  of  questions  to  which  the 
residents  respond.  Ideas  and  comments  are  recorded  during  the 
meetings  and  residents  answer  questionnaires  regarding  the 
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service  priorities  and  needs  of  the  community.  The  results 
of  the  forum  are  publicized  through  the  mass  media,  extensive 
railings  and  contacts  with  organizations  such  as  churches  and 
schools.  Advantages  of  the  forum  are  its  low  cost  and  flexible, 
simple  design.  It  also  enlarges  the  circle  of  informants  to 
include  as  wide  a  cross-section  of  the  community  citizens  as 
possible.  Finally,  it  provides  a  catalyst  for  the  initiation 
of  plans  and  actions  about  the  human  service  needs  in  the 
community.  As  a  result,  a  more  effective,  broader  based 
community  approach  to  needs  assessment,  the  allocation  of 
resources  and  the  establishment  of  priorities  can  take  place. 

The  community  forum  and  survey  approaches  can  be  used  together 
to  elicit  community  views  and  in  combination  they  provide  one 
of  the  most  reliable  mechanisms  for  acquiring  a  convergent 
analysis  of  needs  and  priorities. 

Siegel  and  Attkisson  (1978:  215)  noted  that  the 
approaches  to  community  needs  assessment  have  in  the  past 
not  arisen  from  co-operative  ventures  between  citizens  and 
professional  personnel.  The  result  has  been  as  Coursey 
(1977:  68)  stated:  .  .  . 

a  gap  between  the  programs  of  many  of  our  human 

service  agencies  and  the  needs  of  those  they  are 

designed  to  serve. 

However,  significant  attempts  have  been  made  to  make  community 
needs  assessments  a  more  co-operative  effort  between  human 
service  management  and  those  it  is  designed  to  serve.  In  this 
way,  it  appears  that  such  approaches  are  acquiring  more  of  a 
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community  development  orientation  whereby  citizens  actively 
participate  in  the  decision-making  processes  by  defining 
their  own  felt  needs  and  thereby  assist  in  providing  input  into 
the  evaluation  process. 

Program  Planning  Evaluation 

According  to  Alkin  and  Fitz-Gibbon  (1975:  3)  during 
program  planning  evaluation  the  focus  is  on  first,  the 
provision  of  evaluative  information  that  might  be  used  in 
achieving  the  desired  goals/ob j ectives  and  second,  on  "the 
development  of  the  planning  document  depicting  subsequent 
evaluation  activities  and  describing  in  its  entirety  the 
instructional  system  to  be  utilized".  Organizations  must  plan 
their  activities  within  an  overall  framework  within  which 
decisions  can  be  made  in  such  a  way  that  they  have  a  rational 
relationship  to  each  other.  There  exists  a  variety  of  concepts 
related  to  the  nature  of  the  planning  process  which  range 
from  the  very  traditional,  simple  formats  such  as  straight 
line  planning  to  those  more  complex  types  that  are  more  charact¬ 
eristic  of  social  action  programs  (Carter  and  Wharf,  1973)  such 
as  the  feedback  approach. 

The  following  discussion  will  outline  the  various 
types  of  planning  processes  and  examine  the  nature  and  relation¬ 
ship  of  evaluative  decisions  towards  measuring  objectives  as 
they  exist  within  the  overall  process  in  which  the  program  is 
planned . 
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According  to  E.  Hamilton-Smith  (1974:  15-24)  many 
organizations  plan  programs  in  an  ad— hoc  way  or  through  the 
concept  of  straight  line  planning  (Figure  3) .  This  planning 
process  involves  an  overall  evaluation  or  a  descriptive 
"stocktaking"  before  implementing  new  plans  through  a  "cross- 
sectional  evaluation  whereby  the  examination  of  organizational 
effectiveness  is  undertaken  at  one  particular  point  in  its 
history  (Hamilton-Smith  1974:  16).  This  type  of  evaluation 
provides  the  basis  for  an  organization  to  reassess  its  goals, 
methods  and  priorities,  and  also  to  reformulate  its  planning 
aPPr°ach  in  such  a  way  as  to  provide  for  future  operation  of 
a  "feedback"  nature. 

The  straight  line  approach  is  a  rather  over-simplified 
planning  process  and  rarely  are  social  action  programs  planned 
in  such  a  way.  According  to  Carter  and  Wharf  (1973:  90)  social 
action  programs  are  characterized  by  "cyclical  features"  which 
require  that  earlier  questions  should  be  asked  such  as  why  a 
certain  cycle  had  been  chosen  for  evaluation,  and  allow  the 
similarity  of  program  goals  to  those  of  previous  cycles  to  be 
examined.  Such  evaluation  activities  require  constant  feedback 
between  the  phases  of  the  planning  process  whereby  adjustments 
take  place  between  each  of  the  steps.  In  this  way  steps  do 
not  proceed  sequentially  but  concurrently  (Figures  4  and  5). 

Monitoring  and  Evaluation  in  the  Feedback  Process 

According  to  Hamilton-Smith  (1974:  22)  monitoring  is  a  technique 
used  within  the  overall  process  of  evaluation  through  a  feedback 
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Figure  3  Examples  of  straight-line  formulations  of  the 

planning  process  for  delivery  of  community  programs 
(From  E.  Hamilton-Smith ,  1974). 
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The  feedback  conception  of  the  planning  process 
for  delivery  of  community  programs  (From  Kahn,  1969)  . 
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Figure  5  Evaluation  as  part  of  the  "feedback"  formulation 
of  planning  for  delivery  of  community  programs 
(From  E.  Hamilton-Smith ,  1974). 
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approach.  Monitoring  is  defined  as  the  systematic  collection 
of  information  relevant  to  the  functioning  of  an  organization 
or  program.  Evaluation  in  turn  uses  this  information  to  make 
intelligent  judgements  concerning  the  organization's  or 
program's  effectiveness,  its  goals,  and  similar  issues,  and,  in 
this  way,  the  two  concepts  are  interrelated.  Monitoring  is  an 
integral  part  of  evaluation  in  a  "feedback"  approach  to 
planning  (Figures  4-6) .  The  information  it  produces  must  be 
communicated  to  those  dealing  with  the  process  of  decision¬ 
making  and  program  implementation  and  they  must  assess  it  and 
utilize  it  to  make  better  decisions  or  to  develop  better 
programs.  Monitoring  plays  an  important  role  within  the  "feed¬ 
back"  planning  process.  A  limitation  of  the  feedback  approach 
is  that  it  can  break  down  when  there  are  interruptions  in  the 
communication  process  or  obstacles  arise  which  affect  the 
utilization  of  the  information  which  is  being  communicated. 

Evaluation  and  Interactive  Planning 

In  inter-active  formulation,  (Figure  7)  E.  Hamilton- 
Smith  (1974:  2-23)  states  that  component  of  the  process  shown 
with  being  concerned  with  data  collection  and  analysis  correspond 
to  the  monitoring  and  will  at  the  same  be  collecting  other  data 
as  portrayed  in  (Figure  6) .  Moreover,  this  component  of  the 
process  will  be  constantly  interacting  and  communicating  with 
the  rest  of  the  components.  In  this  way,  the  definition  of 
objectives,  the  statement  of  goals  and  philosophy,  the  formula¬ 
tion  of  planning  and  the  implementation  of  planning  are 
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Figure  6  The  practical  example  of  the  feedback  approach  to 

planning  for  delivery  of  community  programs  (From 
Milton  Keynes,  1970). 
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Figure  7  Planning  as  an  inter-active  process  for  delivery 
of  community  programs  (From  Burton,  1974). 
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influenced  by  the  information  made  available  and  also  the  data 
being  collected  and  the  way  in  which  it  is  analyzed  will,  in 
turn,  be  vulnerable  to  changing  demands  from  the  other  components 
of  the  process.  Such  a  process  of  interaction  creates  problems 
in  many  practical  solutions.  For  example,  data  collection  analysis 
and  evaluation  is  usually  a  time  consuming  process  pursued  over 
a  long  time  span  to  acquire  real  confidence  in  its  judgements. 
Perversely,  an  organization  pursuing  an  inter-active  model  is 
likely  to  be  continually  shifting  its  emphasis  which  consequently 
creates  a  great  practical  difficulty  for  evaluators,  a 
dilemma  we  see  reflected  in  the  words  of  Wildavsky  as  cited  by 
Smith  (1974 :  23) : 

Evaluation  requires  a  certain  amount  of  stability; 
changing  the  program  month  by  month  makes  it 
impossible  to  get  a  fix  on  it  of  sufficient 
duration  to  perform  any  study. 

In  short,  we  have  seen  that  social  action  programs 
are  most  often  planned  according  to  a  more  complex  process 
such  as  the  "feedback"  or  "inter-active"  approaches.  These 
processes  are  more  demanding  and  require  more  careful  scrutiny 
by  the  evaluator  in  order  to  determine  the  nature  of  his 
decisions  towards  measuring  program  objectives. 

The  author  would  like  to  summarize  at  this  point  what 
has  been  found  within  the  input  evaluation  level  that  is 
primarily  characterized  by  providing  the  resource  requirements 
for  the  setting  of  overall  program  objectives.  As  has  been 
seen,  the  nature  of  social  action  programs  can  make  the  setting 
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of  objectives  difficult,  but  this  problem  can  be  alleviated 
greatly  when  objectives  are  defined  in  accordance  with  the 
theoretical  premises  of  the  program  and  tied  in  with  financial 
considerations.  Furthermore,  the  conceptualization  and  measure¬ 
ment  of  objectives  is  performed  with  more  ease  when  done  in 
accordance  with  the  pre-f ormative  evaluation  activities  defined 
as  needs  assessment  and  program  planning  evaluation,  the  former 
of  which  provides  an  important  informational  base  or  input  into 
the  conceptualization  of  objectives  and  the  latter  of  which 
helps  to  identify  the  subsequent  evaluation  activities  to  assess 
objectives  in  accordance  with  the  way  in  which  a  program  is 
planned.  When  these  activities  have  been  performed,  the 
researcher  proceeds  to  the  next  evaluation  level. 

PROCESS  EVALUATION  LEVEL 

The  social  action  program  and  its  overall  objectives 
having  been  set,  the  next  steps  are  to  document  and  measure 
what  goes  on  within  the  various  program  stages.  The  focus 
here  is  on  the  process  evaluation  level.  The  evaluator  must 
determine  whether  or  not  the  services  were  successfully 
implemented  and  whether  they  succeeded  in  achieving  the  stated 
objectives  through  an  examination  of  progress  (formative 
activities) .  A  number  of  considerations  are  critical  at  this 
stage;  first,  there  is  the  need  to  specify  measures,  both 
quantitative  and  qualitative;  second,  how  to  study  them  within 
a  research  design  (e.g.  experimental) ;  and  finally,  how  to 
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synchronize  methods  and  procedures  within  an  overall  paradigm 
(e.g.  goal  based  or  systems  model) .  It  is  these  considerations 
which  are  examined  within  this  section. 

Specification  of  Measures 

Although,  as  Carter  and  Wharf  point  out,  little  has 
been  written  specifically  on  measuring  techniques  for  evaluation 
of  social  action  programs,  there  exists  within  the  available 
literature  some  sources  which  offer  significant  guidelines. 

Carol  Weiss  (1972:  45-50)  offers  one  of  the  best 
reviews  of  the  specification  of  measures  related  to  social 
action  programs.  Weiss  contends  that  the  conceptualization  and 
assessment  of  desired  outcomes  is  done  through  the  soecif ication 
of  output  variables  which  measure  the  effects  upon  the  agencies 
and  larger  systems  and  are  fairly  easy  to  observe  and  quantify. 
However,  more  important  is  the  need  to  assess  outcomes  in  terms 
of  the  effects  upon  program  recipients  such  as  changes  in 
knowledge,  skills,  attitudes,  values  and  personality.  Such 
behavioral  changes  are  difficult  to  assess  as  they  do  not  deal 
with  physical  attributes  which  are  easy  to  see,  touch,  quantify 
or  observe.  The  result  has  been  to  ignore  or  not  develop 
adequate  qualitative  measures  which  according  to  Henderson 
(1974 :  40)  are. .  .  . 

qualitative  social  indicators  which  are  subjective 
measures  of  how  individuals  or  groups  within  the 
population  view  the  functioning  of  and  changes 
within  the  various  subsystems  and  contain  those 
definitions  or  partial  definitions  of  social  indicators 
that  bear  on  the  subjective  or  qualitative  perception 
of  the  situation. 
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The  development  of  sound  qualitative  measures  requires  more 
attention  because  as  Zimberg  (1976:  226)  contends  we  must 
account  for  the  measurement  of  psychological  variables  as 
their  consideration  is  important  in  assessing  the  success  or 
failure  of  a  program  in  affecting  changes  upon  program  recipients. 

Mark  Schneider  (1976:  237)  offers  a  useful  differentia¬ 
tion  between  objective  social  indicators  which  are  aimed  at 
qualitative  changes  in  groups  and  subjective  indicators 
designed  to  measure  the  quality  of  life  in  the  individual.  He 
contends  that  a  confusion  between  the  two  terms  has  led  to  a 
blurring  of  the  physical  and  psychological  aspects  of  life's 
qualities  and  that  efforts  towards  developing  more  adequate 
social  indicators  applicable  for  measuring  qualitative,  in¬ 
tangible  behavioural  changes  such  as  changes  in  attitudes, 
aspirations,  expectations,  and  happiness  levels  are  needed.  In 
short,  when  dealing  with  qualitative  changes  and  subsequent 
measures,  there  is  greater  need  to  apply  judgement  and  less 
scientific  precision  from  the  evaluator.  In  doing  this,  the 
evaluator  is  not  being  less  objective  (Zimberg,  1976)  but 
rather  is  capitalizing  on  the  role  judgement  can  play  in  the 
evaluation  process  (Berk,  Rossi,  1976:  338).  Furthermore,  such 
subjective  indicators  do  assist  in  the  relative  ranking 
of  quantitative  objective  indicators  and  clarify  the  signifi¬ 
cance  of  these  (Henderson,  1974:  40). 

Weiss  (1972:  43)  points  out  that  in  addition  to 
conceptualizing  the  desired  outcomes  is  to  develop  variables 
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whose  measurement  help  to  clarify  the  meaning  of  the  program 
in  two  ways;  first;  they  fill  in  the  details  of  what  the  general 
program  has  outlined;  by  illustrating  the  range  of  elements 
that  are  inherent  in  the  program;  and  second,  they  contribute 
to  the  analysis  of  which  features  of  the  program  work  and  which 
do  not  .  An  analysis  of  program  variables  begins  to  explain 
why  the  program  has  the  effects  it  does.  However,  the  concept¬ 
ualization  of  the  nature  of  a  social  action  program  can  be  a 
difficult  undertaking  due  to  their  complex  nature.  In  order 
to  overcome  this  obstacle  Weiss  contends  that: 

program  variables  should  be  given  more  precision  in 
terms  of  differentiating  between  input,  intervening 
variables  in  order  to  explain  why  the  program  has 
produced  such  effects  then  we  can  begin  to  see  which 
program  factors  are  correlated  with  success  and  thus, 
have  the  basis  for  recommendations  for  future  modifi¬ 
cations  . 

Input  measures  are  determined  by  variations  in 
purposes,  methods,  principles,  staffing,  recipients,  service 
duration,  location,  program  size  or  management  auspices. 

Weiss  also  establishes  criteria  for  the  measurement  of  partici¬ 
pant  input  measures  which  include  such  variables  as  age,  sex, 
socio-economic  status,  race,  length  of  residency,  reasons  for 
participation  and  attitudes  and  aspirations  about  programs. 

Also,  Weiss  stresses  the  need  to  specify  conditions  between 
the  program  inputs  and  outcomes.  He  calls  these  intermediate 
factors  intervening  variables,  (e.g.  frequency  of  staff  turnover), 
which  occur  between  program  inputs  and  outputs  and  can  there¬ 
fore  influence  the  program's  direction. 
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In  short,  through  the  development  of  output  variables, 
to  measure  the  extent  to  which  overall  program  goals  have 
been  achieved,  the  evaluator  is  concerned  too,  with  the 
description  and  measure  of  other  relevant  aspects  of  the 
program  such  as  the  inputs  and  also,  those  factors,  known  as 
intervening  variables  that  mediate  between  inputs  and  outputs. 

Research  Designs 

Once  the  measures  have  been  specified  and  the  data 
have  been  collected  then  the  next  step  is  to  decide  on  how  to 
study  the  program  effects.  In  formulating  the  research  design 
the  evaluator  can  use  the  traditional  controlled  experiment, 
the  less  formalized  quasi-exper imental  designs,  or  move  to 
dynamic,  descriptive  designs  whose  appropriateness  for  handling 
the  nature  of  social  action  programs  will  be  outlined  in 
this  section. 

Logsden  (1975:  32)  offers  a  useful  description  of  the 
experimental  design  which,  briefly,  randomly  selects  a  large 
number  of  candidates  from  a  community  and  assigns  them  to  a 
program  experiment  group  or  a  control  group.  The  program  is 
viewed  as  an  experimental  treatment  and  statistical  tests  are 
utilized  to  establish  whether  the  program  group  achieved  more 
goal-oriented  progress  than  the  control  group  (Figure  8) . 

Rossi  (1979)  maintains  that  the  experimental  design 
is  still  best  for  evaluating  social  action  programs.  He 
contends  that  all  elements  in  both  action  programs  and 
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REFERENCE  POPULATION 


Figure  8 


Experimental  evaluation  design. 
(From  Rochon,  1972). 
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experimental  design  are  present.  That  is,  both  are  under 
someone's  control,  and  therefore  it  is  possible  to  construct 
experimental  and  control  groups.  Moreover,  the  program  is 
usually  only  for  a  portion  of  the  population  which  can  there¬ 
fore  be  duplicated  by  control  groups. 

Donald  T.  Campbell  (1974)  advocates  the  use  of  "quasi¬ 
experiments"  for  research  in  real  world  situations  which 
partially  approximate  experiments  but  in  which  some  significant 
feature  of  the  experiment  is  not  feasible.  Campbell  applies 
the  logic  of  experimental  design  to  these  situations  and  offers 
several  quasi-experimental  designs  which  are  appropriate  under 
different  circumstances.  Although  the  option  of  quasi- 
experimental  design  appears  to  be  more  appropriate  than 
experimental  design  for  social  action  programs,  a  consensus 
from  the  literature  indicates  that  both  designs  are  in¬ 
appropriate  for  social  action  programs  due  to  the  following 
reasons  outlined  by  Logsden  (1975:  33): 

1.  Programmers  may  find  it  very  difficult  to  define 
and  operationalize  goals. 

2.  Randomization  and  control  groups  are  precluded 
since  the  researcher  maintains  minimal  influence 
over  selection  into  the  program. 

3.  Developing  and  pre-testing  goal  indicators  for 
validity  and  reliability  may  not  be  feasible,  if 
standardized  or  pre-tested  indicators  aren't 
available . 

4.  The  study  population  may  be  inaccessible  to 
testing,  biased  against  questionnaires,  and  may 
experience  a  high  dropout  rate  from  the  program 
and  study,  especially  if  the  population  is  of  a 
low  income  nature. 


.  ■ 
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5.  Very  few  controls,  if  any,  may  be  possible  if  the 
evaluation  is  sought  after  the  program  has  been 
completed . 

The  move  from  experimental  design  to  descriptive 
research  designs  appears  to  be  the  most  advantageous  for 
social  action  researchers;  first,  by  incorporating  more  than 
one  stage  of  the  causal  process  into  the  analysis,  the  research¬ 
er  can  more  fully  examine  the  total  social  programming  process 
as  it  occurs;  second,  they  lend  themselves  more  readily  to 
changes  in  the  design  while  an  evaluation  of  it  is  being  carried 
out;  third,  descriptive  designs  capture  the  richness  of  program 
effects,  the  feelings  of  staff  and  participants  and  thus,  make 
the  evaluation  a  co-operative  effort  and  learning  experience 
between  those  intimately  involved  in  the  evaluation  process  as 
well  as  the  potential  consumers  of  the  program. 

An  examination  will  now  be  undertaken  of  the  most 
widely  used  descriptive  research  designs  and  their  techniques 
for  conducting  evaluation  research  in  social  action  programs: 
namely,  the  case  study  approach;  the  survey  of  subjective 
opinions  and  skills;  and  selected  variations  upon  the  group 
process  approach,  such  as  the  transactional  and  adversary 
techniques . 

The  case  study  approach  involves  a  careful  examination 
of  all  letters,  memorandums  and  other  written  material  which 
have  established  procedural  policies  and  which  designate 
policies  and  restrictions  in  resource  allocation.  Key  people 
within  various  levels  of  agencies  may  be  interviewed.  It 
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necessitates  expertise  in  the  designing  of  valid  and  objective 
analytic  infrastructures  and  in  the  carrying  out  of  rigorous 
analysis.  One  of  its  limitations  is  that  many  personnel  have 
neither  the  indication,  time,  nor  expertise  for  such  evaluation 
efforts  (Poplin,  1972) . 

One  source  of  collecting  data  under  the  case  study 
approach  is  through  participant  observation  (Poplin,  1972). 
Participant  observation  is  an  effective  way  to  study  social 
movements,  communities  and  informal  groups.  Unlike  the  social 
survey,  it  does  not  lend  itself  to  large  amounts  of  statistical 
data  nor  is  it  highly  structured.  Through  his  participation 
within  the  community  he  wishes  to  study,  the  researcher  acquires 
data  and  gains  insight  into  the  community  process  and  structure. 
As  a  method,  participant  observation  uses  techniques  of  direct 
observation,  direct  participation,  a  great  deal  of  interviewing 
and  a  wide  variety  of  documents  and  unpublished  records  in¬ 
cluding  statistical  material.  The  primary  advantage  of 
participant  observation  as  a  basic  methodological  approach  is 
that  it  yields  certain  types  of  data  which  can  only  be  discov¬ 
ered  by  the  researcher  who  enters  into  a  continuous  and  close 
interaction  with  his  informants  and  also,  the  researcher  is 
not  restricted  to  a  particular  set  of  questions.  Limitations 
to  the  participant  observation  method  include,  most  notably, 
that  the  researcher  may  unavoidably  influence  the  group  he  is 
observing . 
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A  survey  of  subjective  opinions  and  skills  is  a 
technique  which  provides  the  programmer  with  an  evaluation 
which  can  be  qaickly  and  easily  achieved.  It  provides  a 
readable  assessment  tool  of  the  program's  inputs  and  assets. 

The  evaluation  questions  in  the  survey  are  related  directly  to 
the  experiences  of  the  staff  and  participants.  Objective  tests 
of  the  acquired  skills  can  be  given  before,  after  or  during 
the  program  in  order  to  confirm  the  degree  to  which  skills 
were  developed  and  utilized  and  such  tests  must  include  the 
reactions  of  participants  and  staff  to  the  program  inputs. 

In  this  way,  the  survey  approach  brings  the  evaluation  closer 
to  the  program  participants.  Moreover,  as  Logsden  (1975:  35) 
points  out,  in  the  majority  of  cases,  participants'  opinions 
and  answers  to  questions  are  reliable.  This  is  supported  by 
the  fact  that  when  these  opinions  are  accompanied  by  frequency 
counts  indicating  the  percentage  of  respondents  benefitting 
from  the  program  something  closely  approximating  the  testimonies 
and  rating  occurs. 

A  variety  of  the  group  process  approaches  can  be 
utilized  in  community  action  program  evaluations  at  relatively 
low  cost.  Such  processes  are  easily  manageable  and  flexible 
within  shorttime  spans.  Group  approaches  are  comprised  of  a 
series  of  group  discussions  between  board  members,  staff  and 
participants  in  order  to  review  program  goals,  processes  and 
benefits  based  on  agency  records,  statistical  data  and  people's 
perceptions  of  problems  and  solutions.  By  bringing  together 
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the  evaluator  and  participants,  the  evaluation  process  is 
not  as  vulnerable  to  manipulation  by  either  side  and  all 
actors  gain  primary  experience  in  research  and  assessment 
through  their  direct  involvement  in  the  evaluation  process. 

In  this  way ,  the  evaluations  become  a  learning  experience  for 
all  engaged  and  training  in  the  integration  and  application  of 
an  evaluation  mechanism  is  achieved  for  future  programming. 

Logsden  (1975:  35)  indicates  several  limitations  of 
group  process  approaches:  first,  they  necessitate  that  all 
actors  come  together  as  a  co-operative  unit  which  is  often 
difficult  to  achieve  or  strategically  unwise;  second,  if  the 
program  continues,  then  there  is  a  possibility  that  opinions  and 
agency  records  can  become  augmented  with  case  study  and  survey  data,  and 
thus,  data  can  become  less  accurate  or  manipulated  by  primary 
actors  conducting  the  process;  and  third,  the  reliance  on  people's 
opinions  leaves  room  for  a  questioning  of  their  validity. 

One  type  of  group  process  approach  called  "transactional" 
was  first  advocated  by  Robert  Rippey  (1972).  It  involves 
protagonists,  and  designers  of  the  innovation  plus  a  represen¬ 
tative  group  of  people  likely  to  be  negatively  affected  by 
consequences  of  change.  This  design  is  based  on  a  resistance 
to  change  and  its  threat  to  roles  and  requires  the  evaluator 
to  have  a  great  deal  of  interpersonal  skills  if  he  is  to  employ 
the  strategy  wisely.  The  model's  advantages  lie  in  the  fact 
that  it  involves  both  protagonists  and  antagonists  of  the  change 
and  the  constant  evaluation  of  both  the  unanticipated  and 
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anticipated  consequences.  The  advantages  of  the  Rippey  approach 
according  to  Brack  (1975:  44)  are: 

1.  Formative  evaluation  design  is  improved  through 

the  involvement  of  a  wider  range  of  opinions  and 
values  in  the  evaluation  design. 

2.  Increased  organizational  efficiency  and  greater 
program  benefits  result  because  of  attentiveness 
to  potential  role  threats. 

3.  The  concern  of  the  evaluator  for  human  values  as 
well  as  program  outcomes  places  him  in  a  better 
relationship  with  personnel  involved  in  the 
change,  bringing  greater  honesty  of  interchange 
and  thus  more  valid  data. 

4.  Involvement  of  a  wider  range  of  interested  personnel 
in  evaluation  leaves  a  residue  of  organizational 
and  evaluative  skills  that  are  of  potential  use 

for  the  organization,  persisting  beyond  either 
the  termination  or  the  solidification  of  the 
original  change. 

Harriet  Talmage  (1975:  39)  has  applied  the  transactional 
approach  towards  evaluating  local  school  and  community  pro¬ 
grams,  and  found  it  to  be  more  effective  than  evaluation 
models  used  in  the  past  as  it  "weds  group  processes  with 
program  development  and  evaluation  activities".  Talmage 
concludes  that  the  evaluation  consultant  must  make  the  skills 
of  the  transactional  approach  available  to  the  people  involved 
in  these  programs  and  thereby  give  them  a  more  active  role  in 
the  evaluation  design. 

Another  type  of  group  process  approach  is  the  adversary 
evaluation.  In  the  adversary  approach,  advocated  by  Kevine 
(1973)  and  Kourlisky  (1973),  much  has  been  based  on  the  work 
of  Rippey' s  transactional  model.  Levine's  adversary  approach 
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is  based  on  a  legal  orientation.  That  is,  an  adversary  acts 
as  a  "cross  examiner"  of  the  evidence  sources  provided  by  the 
primary  investigator.  The  sources  of  evidence  are  compiled 
and  classified  by  a  designated  group. 

The  Kourilsky  model  uses  both  an  affirmative  and  a 
negative  evaluator  and  each  is  responsible  for  preparing  a 
case  for  and  against  a  proposal  (Figure  9) .  The  information 
from  both  sides  is  thus  provided  to  the  decision-maker  who 
interacts  with  the  two  evaluators  before  the  final  decision 
is  drawn  up.  Kourilsky  (1973:  5)  points  out  that  "adversary 
evaluation  receives  the  benefits  of  both  the  dialective  and 
the  legal  process  and,  as  a  result,  a  better  and  more  rational 
decision  may  result".  In  this  way,  the  decision-maker  takes 
in  a  great  deal  of  information,  primarily  because  both  sides 
are  represented.  In  addition,  during  the  confrontation,  both 
advocates  try  to  cover  most  of  the  factors  supporting  their 
decisions.  The  quality  of  evidence  tends  to  be  better  as  the 
adversary  relationship  encourages  both  sides  to  provide  the  best 
evidence  that  supports  his/her  viewpoint. 

In  summary,  dynamic  descriptive  designs  rather  than 
the  experimental  design  appear  to  be  the  most  feasible  and 
advantageous  for  evaluating  social  action  programs.  They  lend 
themselves  more  readily  to  changes  in  the  design  of  the  program 
while  it  is  being  conducted.  As  has  been  noted,  program 
administrators,  quite  wisely,  look  for  feedback  and  alter 
programs  in  midstream  on  the  basis  of  such  feedback.  A  strict 
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Figure  9  Adversary  evaluation  flow  chart  (From  Kourilsky, 

1973) 
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experimental  design  may  be  jeopardized  by  such  changes,  whereas 
the  more  descriptive  designs  benefit  from  such  changes  and 
allow  the  researcher  to  more  fully  understand  the  social 
programming  process  as  it  actually  happens  (Voth,  1975) .  It 
should  also  be  pointed  out  that  the  choice  of  research  design 
has  direct  implications  for  the  selection  of  the  overriding 
model  which  illustrates  in  a  simplified  form  how  a  program's 
methods  and  procedures  should  operate  during  the  process  evalu¬ 
ation  level.  Such  a  paradigm  is  useful  for  determining  the 
degree  to  which  the  underlying  concepts  are  understood  and 
should  be  adopted  in  accordance  with  specifics  of  the  program 
to  which  it  is  to  be  applied. 

Study  Models 

It  has  been  stated  that  the  choice  of  research  design 
has  direct  implications  on  the  selection  of  the  overriding 
model  which  illustrates  in  a  simplified  form  how  a  program's 
methods  and  procedures  should  operate  during  the  process 
evaluation  level.  Such  a  paradigm  is  useful  to  the  degree  to 
which  the  underlying  concepts  are  understood  and  should  be 
adopted  in  accordance  to  specifics  of  the  program  to  which 
it  is  to  be  applied. 

There  are  two  major  trends  in  the  literature;  first, 
an  evaluation  based  on  a  deductive  goal  based  model  that  is 
used  in  accordance  with  the  experimental  design;  second,  an 
indicative,  descriptive  approach  based  on  a  systems  model 
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As  both  the  goal-model  approach  and  the  systems  model  appear 
so  frequently  in  the  literature  and  both  techniques  have  been 
widely  operationalized  in  evaluation  research,  it  becomes 
imperative  to  examine  the  strengths  and  weaknesses  of  each 
in  order  to  determine  their  applicability  for  evaluating 
social  action  programs. 

The  goal  model  approach  (Figure  10)  has  been  extensively 
examined  in  the  literature  (Freeman  and  Sherwood  (1977), 

Denton,  (1975),  Schulberg  and  Baker  (1977),  Thompson  (1975), 
Seutsher  (1976)  and  Rutman  (1976).  It  has  been  widely  accepted 
and  assumes  the  basic  concept  expressed  by  Thompson  (1975:  14)... 

evaluation  is  the  measurement  of  movement  towards  a 

goal . 

Schulberg  and  Baker  (1977  :  54-63)  point  out  that  the  goal- 
model  approach  assumes  that  goals  can  be  readily  identified  and 
subdivided  into  various  levels  or  subgoals  to  which  appropriate 
methodologies  and  criteria  can  be  applied  so  as  to  measure  effect¬ 
iveness  in  terms  of  efforts  (inputs  and  outputs)  and  performance 
(progress  towards  goal  attainment) .  It  is  a  very  structured 
evaluation  process  beginning  at  the  lowest  goal  levels  and 
moving  upwards,  concerning  itself  with  the  degree  of  success 
in  reaching  a  specific  objective  which  can  be  evaluated  and 
modified  in  isolation  from  other  goals  established  by  the 
organization.  Thus,  the  goal-model  approach  does  not  highlight 
the  interrelation  between  goals  and  the  ways  in  which  modifica¬ 
tion  of  one  goal  can  become  constrained  by  others  in  the  program's 
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Figure  10 


An  example  of  the  goal-model  approach 
(From  Carter  and  Wharf,  1973). 


58 


process  characteristics.  Schulberg  and  Baker  point  out  that 
a  further  limitation  of  the  goal  attainment  model  is  that  it 
is  relatively  detached  from  any  concern  with  the  techniques 
of  implementing  findings.  This  weakness  is  due  to  its  close 
approximation  with  the  experimental  model;  that  is,  in  trying 
to  obtain  objectivity,  the  researcher  relies  on  the  administra 
tor  for  a  statement  of  goals  without  considering  the  fact  that 
many  organizational  goals  were  never  intended  to  be  formalized 
In  this  instance  the  program  administrator  will  be  scarcely 
disturbed  when  the  researcher  finds  that  the  organizational 
goals  are  not  being  met  and  according  to  Schulberg  and  Baker 
(1977:  57) .  .  . 

never  having  meant  to  attain  the  goals  studied 
by  the  researcher,  the  administrator  sees  no 
need  to  alter  his  programs  to  accommodate  the 
findings  of  the  research.  The  program  evaluation 
has  little  impact  upon  the  organization  since  the 
researcher  had  little  understanding  of  the  admini¬ 
strator's  purpose  in  participating  in  the  study. 

Deutscher  (1976:  250)  suggests  that  evaluator  and 

administrator  should  negotiate  over  defining  program  goals 

in  order  to  overcome  confusion.  Deutscher  differentiates 

between  original,  formal  goals  and  actual  or  real  goals, 

the  latter  of  which  become  a  "bastardized"  version  of  the 

former  during  the  planning-implementing-evaluation  process. 

Through  negotiations,  the  researcher  and  administrator  can 

discuss  and  define  which  goals  they  wish  to  examine  and 

measure.  Administrators  can  reach  an  agreement  on  what  the 

"actual  or  real  goals"  are  that  need  to  be  assessed.  In  this 
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way  evaluating  spurious  goals  and  utilizing  invalid  measures 
is  prevented.  Denton  (1977)  is  also  concerned  with  the 
measurement  and  acceptability  of  goals  which  are  often  contra¬ 
dictory  concepts;  imprecise  or  too  difficult  for  a  program 
to  meet;  program  goals  are  made  precise  then  little  concensus 
exists  on  their  acceptability. 

Further  limitations  of  this  model,  as  Carter  and  Wharf 
(1973:  45)  point  out  is  that  the  goal-model  approach  does  not 
describe  accurately  all  program  variables  and  it  is  difficult 
to  make  the  important  program  variables  (input,  intermediate, 
and  output)  operational.  First,  input  measurements  require 
a  close  scrutiny  of  program  proposals  and  organizational  charts, 
neither  of  which  is  accommodated  for  in  this  approach;  second, 
intermediate  variables,  which  include  the  difficult  to  measure 
attitudinal  and  cognitive  changes  that  are  of  major  concern 
in  social  action  programs,  are  not  highlighted;  finally,  output 
measures  require  a  close  observation  of  work  and  training 
sessions  which  is  also  not  possible  under  this  approach. 

Several  modifications  of  the  goal -model  approach  have 
been  proposed  in  an  effort  to  improve  its  usefulness  for  evalu¬ 
ation,  some  of  which  will  now  be  examined. 

James  (1961)  described  a  less  rigid,  circular  version 
of  the  evaluation  process  which  began  with  goal  setting,  went  on  to 
measuring  the  goal,  collecting  data  and  appraising  it  in 
relation  to  goal  effects  and  finally,  the  initial  goal  is 
modified  on  the  basis  of  the  data  collected.  A  weakness  of 


. 
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James  modified  goal  approach  according  to  Shulberg  and  Baker 
(1977:  59)  is  that  there  is  no  sound  way  that  the  circle  of  the 
evaluation  process  can  be  closed  and  that  usually  what  had 
previously  been  a  reciprocal  relationship  of  co-operation 
between  evaluator  and  administrator  breaks  down  at  the  point 
of  goal  modification. 

Another  goal  approach  modification  includes  Michael 
Scriven's  (1972)  "Goal  Free  Evaluation"  which  is  concerned 
with  the  side  effects  or  "unanticipated"  events  which  should 
be  separated  from  goals.  He  is  not  advocating  a  non-goal 
approach  but  rather  is  trying  to  improve  the  approach  by  advo¬ 
cating  that  a  sound  evaluation  should  be  able  to  determine  both 
intended  goals  and  unintended  effects  of  a  program.  "Goal 
Free  Evaluation"  is  supported  by  Stuff lebeam  and  Kneller  (1972) 
but  as  Brack  (1975)  points  out  certain  limitations  exist  in 
Scriven's  approach  which  inhibit  an  evaluator  from  becoming 
overly  optimistic  about  its  utility.  That  is,  goal  free  evalu¬ 
ation  could  be  utilized  as  an  excuse  for  not  being  able  to 
state  goals;  it  may  not  provide  a  focus  for  data  collection; 
and  there  could  be  a  lack  of  existing  trained  evaluators  who 
are  able  to  impose  their  own  goals  upon  a  program  in  order  to 
carry  out  a  thorough  evaluation  (Brack,  1975) . 

Goal  attainment  scaling,  first  formulated  by  Sherman 
and  Kiresuk  (1968),  rests  in  the  general  tradition  of  the  goal- 
model  approach,  but  in  addition  to  the  general  form  this 
approach  just  described  it  makes  use  of  quantitative  techniques 
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and  analysis.  As  Kiresuk  and  Lund  describe  it  (1978:  342) 

"goal  attainment  scaling  improves  the  situation  by  placing 
the  target  goal  in  the  center  of  a  range  of  possible  outcomes 
from  "most  unfavourable"  and  "less  than  expected"  on  the  one 
end ,  the  "expected  outcome"  in  the  middle,  and  the  "more  th  an. 
expected"  and  "best  anticipated"  on  the  other  end.  This  is 
incorporated  in  the  left  column  of  a  "goal  attainment  follow¬ 
up  guide".  Goal  attainment  scaling  (Coursey  1977:  49)  has  become 
a  widely  used  quantitative  technique  which  measures  the  extent 
to  which  a  client  has  achieved  her/his  own  treatment  goals. 

Nancy  Wilson  (1977)  has  summarized  how  computers  can  be  usefully 
utilized  during  the  retrieval  system  of  data  collection  for  goal 
attainment  scaling.  Finally,  Miller  (1975)  and  Webber  (1975) 
have  integrated  goal  attainment  techniques  into  a  wider  manage¬ 
ment  information  system.  Also  Zinober  (1974)  and  Miller  (1975) 
have  developed  goal-orientated  evaluations  for  measuring  effects 
upon  children  engaged  in  educational  programs. 

The  major  alternative  to  the  goal  model  is  the  systems 
approach  (Figure  11) .  The  system-model  approach  recognizes 
more  explicitly  than  the  goal  model  approach  that  there  exists 
a  complex  network  of  program  goals  (Thompson,  1975) .  According 
to  Coursey  (1977:  51)  the  systems  model  must.  .  . 

achieve  its  goals  and  subgoals;  effectively  coordinate 
organizational  subunits;  acquire  and  maintain  necessary 
resources;  and  adapt  the  organization  to  the  environ¬ 
ment  and  to  its  own  internal  demands. 

In  this  way  the  organization  is  seen  as  a  multifunctional  unit. 
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An  example  of  the  systems-model  approach 
(From  Carter  and  Wharf,  1973). 
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The  system-model  is  characterized  by  being  an  open  system 

which  interacts  with  its  environment.  It  constantly  collects 

information  about  the  environment  (e.g.  administrative  aspects) 

and  behavioral  reactions  to  events  happening  within  and  between 

component  parts  and  uses  this  information  as  the  means  for 

channelling  adaptations  or  directions  toward  the  system's  goals. 

The  concept  most  commonly  borrowed  by  evaluators  from 

the  system  approach  is  the  input-throughput-output-feedback 

notion.  Borrowing  from  Ryan's  summary  (1973:  50-57)  inputs 

are  the  components  which  receive,  store,  or  take,  in  energy 

in  the  form  of  material  or  information  such  as  money,  time, 

people,  equipment,  effort  or  information:  throughput  is  the 

convertor  processor  or  means  of  assimilating  or  turning  the 

energy  received  into;  output,  the  result,  product  or  outcomes; 

feedback  is  the  regulating  feature  which  serves  as  a  monitor 

and  evaluator  to  ensure  that  planned  goals  are  reached.  In 

this  way,  as  Ryan  (1973:  53-54)  concludes: 

In  an  open  system,  the  whole  is  considered  in  terms 
of  flow.  The  flow  process  involves  input  as  the 
energizer  (or  starting  force) ,  throughput  as  the 
process  activity,  and  output  as  the  result  of  the 
action.  Feedback  is  that  part  of  the  flow  which 
serves  as  a  control  by  continually  monitoring  and 
evaluating  all  component  actions  and  interactions. 

The  flow  process  is  cyclic  in  operation;  it  strives 
for  balance  and  seeks  to  sustain  the  life  of  the 
system. 

This  input-throughout-output-f eedback  scheme  has  been  widely 
used  for  systems  engineering  in  areas  of  information  theory 
and  cybernetics.  Also,  this  form  of  systems  design  has  been 
widely  utilized  in  the  business,  education  and  health  fields. 


. 
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It  appears  that  although  the  system -model  approach  is 
more  demanding  than  the  goal  approach,  it  permits  a  better 
analysis  of  social  action  programs  whose  aims  are  not  confined 
to  limited  objectives  but  include  new  ways  of  utilizing  avail¬ 
able  resources.  In  this  way,  by  incorporating  more  than  one 
stage  of  the  causal  process  into  the  analysis,  a  researcher 
may  more  fully  explore  and  conceptualize  the  total  change 
processes  initiatied  by  social  action  programs.  Furthermore, 
through  a  systems  approach,  an  evaluator  can  aim  at  the  present 
stage  of  program  development  which  necessitates  a  different 
set  of  evaluation  questions  to  be  asked  at  different  program 
stages  based  on  questions  of  how  effectively  program  objectives 
have  been  met  in  terms  of  effort,  effectiveness  and  efficiency. 
This  type  of  analysis  has  been  termed  differential  evaluation 
(Tripodi,  1971:  41). 

In  summary,  the  process  evaluation  level  appears  to 
be  the  most  demanding  and  time  consuming  evaluation  level.  It 
covers  a  wide  range  of  complex  elements.  First,  there  is  the 
specification  of  quantitative  and  qualitative  measures,  the 
latter  of  which  is  more  difficult  than  the  former  as  it  deals 
with  assessing  intangible,  behavioral  changes  and  as  they  appear 
to  be  the  target  of  many  social  action  programs,  the  researcher 
must  make  a  great  deal  of  effort  towards  developing  such  sound 
measures.  Second,  once  the  measures  have  been  specified  and 
the  data  collected,  the  researcher  chooses  the  appropriate 
design  in  order  to  study  the  program's  progress.  Here,  it 
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appears  that  the  descriptive  research  design  in  the  majority 
of  cases,  is  most  conducive  towards  a  holistic  assessment  of 
the  effects  of  social  action  programs.  Third,  the  methods 
and  procedures  of  descriptive  research  design  are  best  synchron¬ 
ized  in  an  overriding  study  model  or  paradigm  known  as  the 
system-model  approach. 

Process  evaluation  appears  to  be  the  primary  focus  of 
most  social  action  programs.  Fitz-Gibbon  and  Alkin  (1975:  11) 
have  summarized  why  it  is  often  impossible  to  reduce  process 
objectives  to  outcomes: 

1.  The  objectives  of  most  action  programs  are  focussed 
on  assessing  difficult  areas  such  as  changes  in  attitudes 

and  values  which  need  time  to  become  effective  and  therefore 
defy  "short-run"  solutions,  such  as  is  obtained  through  out¬ 
come/end  product  evaluation. 

2.  The  striving  for  intangible  or  distant  outcomes 
cannot  be  measured  immediately  and  no  proxy  measures  of  proven 
validity  and  reliability  are  available. 

3.  End  product  evaluation  studies  are  inappropriate 
in  the  beginning  stages  when  a  program  usually  reflects 
unstable  characteristics,  and  difficulty  in  specifying  objectives. 
And,  often  when  changes  need  to  be  made,  they  can  be  temporarily 
disruptive  to  an  evaluation  and  force  the  evaluator  to  make 
changes  to  what  he  perceives  to  be  the  desired  outcome. 

4.  An  outcome/end  product  evaluation  fails  to  account 
for  oartial  or  unanticipated  results  which  are  important 
variables  that  must  be  accounted  for  in  assessing  the  success 
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or  failure  of  a  program. 

In  the  last  regard.  Carter  and  Wharf  (1975:  65)  noted 
that  the  Westinghouse  evaluation  of  Head  Start  programs  in  the 
United  States  failed  to  emphasize  that  important  secondary 
consequences  arose  from  the  process  which  affected  the  health, 
nutrition  and  community  objectives  of  the  program  and  these 
partial  results  became  responsible  for  making  it  .  .  . 

the  single  largest  health  services  delivery 

system  to  needy  children  in  the  nation. 

Hence,  it  appears  that  it  is  often  impossible  to  reduce 
process  objectives  to  an  outcome/product  study  and  the  evalu¬ 
ator  who  insists  on  trying  to  get  outcome  when  the  program 
audience  is  determined  on  assessing  process  goals  fails  to 
see  in  the  words  of  Alkin  and  Fitz-Gibbon  (1975:  11)  "that  the 
proof  of  the  pudding  is  always  in  the  growth  and  never  in  the 
eating".  In  this  context,  it  appears  that  the  evaluator  of 
social  action  programs  must  focus  on  examining  the  nature  of 
a  program  revealed  through  the  process  evaluation  level  and 
if  he  is  just  concerned  with  examining  the  end  product  he  may 
fail  to  ask  the  appropriate  evaluative  questions.  Moreover, 
it  appears  that  the  process  evaluation  level  deals  with  those 
elements  w hich  provide  important  decision-making  areas  between 
the  input  and  product  evaluation  levels  in  order  for  a  more 
comprehensive  evaluation  to  be  undertaken. 


■ 
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PRODUCT  EVALUATION  LEVEL 

Product  evaluations  refer  to  how  well  the  objectives 
of  a  program  have  been  met  and  this  level  is  assessed  through 
two  activities  called  documentation  and  outcome  (summative 
evaluations).  As  the  preceeding  section  indicated,  outcome 
studies  focused  on  examining  the  end  product  are  not  always 
feasible  with  regard  to  social  action  programs.  Nevertheless, 
a  familiarization  with  the  various  types  and  techniques  of 
outcome  studies  are  useful  to  the  practitioner  trying  to  acquire 
an  amount  of  expertise  in  evaluation. 

The  following  section  will  focus  on  methodologies  of 
the  product  evaluation  level  which  refer  to  evaluating  how  well 
the  objectives  of  a  program  have  been  met.  A  brief  description 
of  an  outcome  model  will  be  outlined  and  followed  by  an  examin¬ 
ation  of  two  major  techniques  towards  documenting  effects 
under  such  an  approach;  the  planning  and  management  methodolo¬ 
gies  (e.g.  decision-theoretic  approach  and  cost  analytic 
models) . 

Outcome  Models 

According  to  Carter  and  Wharf  (1973:  41-43)  outcome 
models  determine  the  degree  to  which  client  improvement  can  be 
attributed  to  program  intervention  and  their  major  focus  is 
upon  changes  in  global  adjustment  in  the  individual  and  is  an 
outgrowth  of  the  classic  experimental  research  design.  Some 
measures  include  self-ratings,  peer  ratings,  mood  scales, 
personality  inventories  and  assessment  interviews. 
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Hargreaves  and  Attkisson  (1978)  have  documented  how 
outcome  evaluation  is  an  advanced  stage  of  program  evaluation 
which  makes  an  important  contribution  to  program  management. 

Planning  and  Management  Methodologies- 

Decision  Theoretic  Approach 

Evaluation  procedures  founded  in  management  perspect¬ 
ives  and  models  have  usually  focused  on  the  planning ,  implemen¬ 
ting,  and  decision-making  aspects  of  programming.  A  model  of 
program  decision-making  which  has  received  a  great  deal  of 
attention  is  the  decision-theoretic  approach  (Guttentag  1973). 
This  approach  according  to  Guttentag  (1973:  140)  offers  a 
conceptual  framework  which  "links  inferences  about  states  of 
the  world,  the  values  of  decision-makers  and  decisions''.  Thus, 
the  decision-theoretic  approach  focuses  on  frequent  program 
decisions  and  the  values  or  "utilities"  of  the  decision-makers 
and  can  be  used  by  agencies  both  in  planning  and  evaluation 
which  are  "continuous  and  are  integrated"  (Guttentag  1977:  78). 
Although  the  experimental  design  can  be  utilized  on  occasion 
in  this  approach  it  emphasizes  using  alternative  designs  (e.g. 
descriptive) in  light  of  the  fact  that  the  classical  experimental 
model  has  not  always  been  conducive  to  meeting  the  needs  of 
decision-makers . 

Cost  Analytic  Models 

As  Coursey  (1977:  53)  pointed  out  there  has  been  a 
great  deal  of  resistance  to  cost-based  evaluations  by 
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administrators  as  they  do  not  like  to  justify  human  and  social 
services  in  cost  terms.  However,  pressure  from  funders  and 
governments  has  increasingly  caused  evaluation  to  be  undertaken 
of  the  fiscal  dimensions  of  programs.  This  model  can  assume 
a  number  of  approaches  including  cost  accounting,  cost-benefit 
analysis,  cost-effectiveness  analysis,  cost  utility  analysis 
and  operations  research.  Each  of  these  approaches  has  been 
well  documented  by  Robert  Coursey  (1977:  53-54),  who  sees 
many  limitations.  Most  important  however,  is  the  fact  that 
there  is  an  exclusive  focus  on  cost  which  can  distort  process 
and  value  considerations.  For  this  reason,  cost  analytic 
models  do  not  lend  themselves  readily  to  improving  economic 
efficiency . 


CONTEXT  EVALUATION  LEVEL 

In  order  to  complete  the  research  framework,  issues 
relating  to  the  context  must  be  considered  as  the  evaluation 
hypotheses,  and  techniques  within  an  evaluation  study  are 
dependent  upon  contextual  elements  revolving  around  and  within 
the  program  (Longest,  1975).  Furthermore,  the  nature  of  con¬ 
textual  dimensions  helps  to  provide  a  more  objective  assessment 
for  the  input,  process  and  outcome  levels  which  we  have  pre¬ 
viously  examined.  This  section  will  examine,  first,  the  organ¬ 
izational  context  within  which  the  program  is  planned, 
implemented  and  evaluated  and  second,  how  the  socio-political 
environments  affect  the  nature  of  the  evaluation  process. 
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Organizational  Context 

Wildavsky  (1972)  states  that  the  "ideal"  organization 
would  be  self  evaluating.  Unfortunately,  such  an  ideal  rarely 
exists  and  the  need  for  evaluation  researchers  continues. 

E.  Hamilton-Smith ' s  (1974)  discussion  of  the  organizational 
aspects  of  evaluation  contains  a  valuable  insight  into  the 
researcher-practitioner  constraints  which  often  arise  during 
the  evaluation  process.  E.  Hamilton-Smith  (1974  :  10)  states  .  .  . 

The  result  of  having  evaluative  studies  that  are 

carried  on  during  the  life  cycle  of  a  program  is 

that  evaluators  and  program  staff  must  live 

(uneasily,  as  we  shall  see)  side  by  side. 

From  a  review  of  the  literature  it  appears  that 
researcher-practitioner  conflicts  can  arise  from: 

1.  Differences  in  personality. 

2.  Differences  in  career  patterns  (the  researcher 
is  often  awarded  a  higher  status  based  on  his  academic  back¬ 
ground  which  creates  resentment  and  feelings  of  rivalry  in 
the  practitioner) . 

3.  Interference  with  the  administration  routine,  as 
the  collection  of  data,  interviews  and  questioning  are  often 
very  time  consuming  processes. 

4.  Incongruent  ideas,  goals,  values  and  frames  of 
reference,  whereby  the  researcher's  use  of  new  evaluation 
techniques  and  dedication  towards  attaining  objectivity 
reflects  a  questioning  of  the  practitioner's  techniques. 

5.  The  very  nature  of  an  evaluation  reveals  facts, 
thus  reducing  the  privacy  and  security  of  some  administrator's 
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jobs . 

There  has  been  considerable  debate  surrounding  the 
decision  to  use  an  internal  or  external  evaluator  in  order 
to  prevent  many  of  the  researcher-practitioner  conflicts. 

The  advantages  of  utilizing  an  evaluator  from  within  the 
organization  are;  that  there  is  usually  an  easier  acceptance 
of  results  from  others  within  the  organization  than  from 
outsiders,  thereby  ensuring  that  changes  are  more  likely  to  be 
implemented;  and  that  an  inside  evaluator  is  more  knowledgeable 
about  every  aspect  of  the  program  and  what  influences  its 
character  and  direction.  However,  inevitably,  an  internal 
evaluator  falls  prey  to  blending  his  biases  into  the  evaluation 
process.  A  specialist  from  outside  can  bring  knowledge  and 
resources  unknown  to  the  agency  in  order  to  solve  problems  of 
evaluation,  he  is  likely  to  be  more  objective  than  program 
personnel,  and  he  can  utilize  the  knowledge  he  has  learned 
from  having  studied  other  programs.  However,  program  personnel 
opposed  to  change  can  easL]y  blame  an  outsider  for  misinter¬ 
preting  facts.  Also,  an  outsider  can  easily  overlook  some 
important  factors  or  hold  biases  upon  which  he  makes  evaluation 
studies.  However,  whether  an  internal  or  external  evaluator 
is  used  many  of  the  problems  inherent  within  the  organizational 
environment  can  be  alleviated  if  the  following  preconditions, 
as  summarized  by  E.  Hamilton  Smith  (1974:  12-14),  are  met: 

1.  Those  responsible  for  the  policy  and  administration 
of  programs  must  be  fully  convinced  of  the  need  for  evaluation; 
must  agree  on  the  purpose  of  evaluation;  must  agree  upon  the 
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uses  and  possible  consequences  of  the  evaluation;  and  must  be 
fully  involved  in  making  the  decision  that  evaluation  will 
be  a  part  of  their  program. 

2.  Organizational  arrangements  must  be  made  to  ensure 
objectivity  together  with  appropriate  resources  and  knowledge. 

This  will  normally  mean  the  involvement  of  external  persons 

in  the  evaluation  process. 

3.  Where  external  personnel  are  involved  in  the 
evaluation  process,  all  steps  of  evaluation  including  design/ 
data  collection  and  data  analysis,  should  be  shared  between 
external  and  internal  personnel  so  that  any  artificial  distinc¬ 
tion  between  evaluator  and  evaluated  is  diminished. 

4.  In  evaluating  any  service  program,  the  recipients 
of  the  service  should  be  involved  in  the  evaluative  process 
as  far  as  possible. 

5.  Those  who  are  involved  in  an  evaluation  process 
should  share  with  each  other  their  basic  assumptions  and  the 
values  underlying  their  thinking  so  that  the  influence  of 
these  upon  perceptions  of  data  and  upon  action  recommendations, 
arising  out  of  data,  will  be  known. 

Socio-Political  Context 

The  organizational  structure  with  its  inherent  researcher- 
practitioner  conflicts  is  not  the  only  area  affecting  the 
conduct  of  evaluation  research.  The  socio-political  environ¬ 
ment  within  which  the  program  exists  also  creates  problems. 
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That  is,  evaluation  research  dees  not  occur  in  a  static  environ¬ 
ment  as  social  action  programs  take  place  in  constantly  shifting 
environments  which  often  meet  unanticipated  events  (Weiss,  1972). 
For  example,  changes  in  clientele  or  community  change  can 
lead  to  subtle  changes  in  program  activities  and  contents. 

In  the  event  that  consequences  arise,  the  evaluator  should  be 
prepared  to  perceive,  handle  and  interpret  changes  accurately 
and  skillfully.  This  can  be  achieved  through  constant 
observation  and  by  examining  any  existing  records  of  evaluations 
conducted  during  previous  program  phases  which  may  provide 
insight  into  any  anticipated  difficulties.  In  these  ways  the 
evaluator  will  be  better  prepared  to  make  "on  the  spot" 
decisions  concerning  how  to  redefine  his  evaluation  scheme. 

The  political  context  within  which  rational  evaluations 
take  place  can  create  constraints  on  the  evaluator  in  three 
major  ways.  First,  assumptions  made  on  what  to  evaluate  are 
political  in  nature  (e.g.  within  the  agency).  Second,  evalua¬ 
tions  as  part  of  the  decision-making  process  inevitably  enter 
into  the  larger  legislative  and  bureaucratic  systems  whereby 
their  formation  and  implementation  are  subjected  to  political 
pressures  (i.e.  funding).  And  finally,  the  political  context 
can  affect  the  dissemination  of  evaluation  results.  That  is 
once,  having  made  it  past  the  lower  political  decision-makers, 
evaluation  reports  often  find  themselves  "gathering  dust"  on 
the  shelves  in  provincial  or  federal  research  departments. 


Brandi  (1978:  7),  in  an  effort  to  link  the  academic 
and  political  worlds,  suggests  that  evaluators  should  become 
more  forceful  in  the  political  arenas  in  order  to  make  their 
results  more  known  and  more  widely  disseminated.  In  this  way 
they  will  not  be  perceived  by  politicians  as  just  another 
interest  group  lobbying  for  attention. 

SUMMARY 

The  important  factors  which  have  emerged  from  this 
critical  appraisal  of  evaluation  literature  from  social  develop¬ 
ment  fields  related  to  community  development  can  be  summarized 
in  the  following  way. 

1.  Evaluation  research  activities  of  social  action 
programs  are  complex  and  there  appear  to  be  four  evaluation 
levels  namely,  input,  process,  product  and  context,  whose 
relationship  to  the  activities  in  the  preformative ,  formative 
and  summative  evaluation  stages  cover  the  major  areas  of 
decision-making  faced  by  the  evaluator.  Consequently,  each 
of  the  levels  should  be  integrated  into  a  framework  in  order 
to  undertake  a  comprehensive  evaluation. 

2.  There  appears  to  be  some  disagreement  over  the 
validity  and  reliability  of  evaluation  studies.  Yet,  the 
majority  agree  that  without  evaluation,  social  action  programs 
become  subject  to  a  variety  of  pressures  from  which  they  must 
demonstrate  some  way  of  communicating  their  value. 
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3.  The  setting  of  objectives  is  often  a  problematic 
task  due  to  the  complex  nature  of  social  action  programs.  Yet, 
this  task  can  be  somewhat  alleviated  when  done  in  reference 

to  the  theoretical  premises  of  the  program.  More  importantly 
the  evaluator  can  receive  important  guidelines  in  conceptu¬ 
alizing  and  measuring  program  objectives  when  he  engages  in 
needs  assessment  and  program  planning  evaluation  activities. 

4.  There  appears  to  be  an  unresolved  issue  over  sound 
qualitative  measures  which  are  difficult  to  develop  and  yet 
remain  as  important  as  quantitative  measures  in  order  to  assess 
intangible,  behavioural  changes  such  as  changes  in  knowledge, 
skills,  attitudes  and  aspirations  which  are  the  target  of  many 
social  action  programs. 

5.  It  appears  that  the  move  from  experimental  to 
descriptive  research  designs  is  most  advantageous  for  the  socia 
action  researcher.  When  these  are  used  in  conjunction  with 
the  system-model  approach,  the  evaluator  can  assess  more 
wholistically  the  processes  of  change  induced  by  social 
action  programs. 

6.  The  primary  emphasis  appears  to  be  on  the  process 
evaluation  level  as  many  action  programs  being  assessed  arc 
not  at  the  stage  where  an  outcome/end  product  evaluation  is 
appropriate.  Moreover,  an  investigation  of  process  helps  to 
reveal  important  partial  results  which  disclose  the  failure 
or  success  of  a  program. 
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7.  As  social  action  programs  take  place  in  turbulent 
settings,  their  direction  can  be  influenced  by  contextual 
factors  such  as  organizational  influences;  which  often  lead 
to  researcher-practitioner  constraints,  or  political  factors 
which  affect  the  proper  dissemination  of  evaluation  results. 

In  conclusion,  having  acquired  a  familiarity  with  the 
evaluation  methodologies  of  applied  social  action  research, 
their  relationship  to  the  major  elements  and  issues  constitu¬ 
ting  the  four  major  evaluation  levels  (input,  process,  product, 
and  context)  and  corollary  evaluation  activities  (pre-f ormative , 
formative,  summative)  it  follows  that  an  appraisal  of  evalu¬ 
ation  literature  related  to  community  development  should  be 
undertaken  in  order  to  determine  their  relationship  and 
applicability  to  this  field. 


CHAPTER  3 


A  CRITICAL  APPRAISAL  OF  EVALUATION  LITERATURE 
RELATED  TO  COMMUNITY  DEVELOPMENT 

The  material  in  this  chapter  will  be  organized  into 
two  parts.  The  first  part  entitled  "Evaluation  as  an  Art" 
will  examine  the  origins  of  evaluation  in  community  develop¬ 
ment.  From  this  historical  backdrop  will  be  depicted  first, 
the  nature  and  role  of  evaluation  in  the  early  history  of 
community  development  which  began  in  the  late  1950's  and  ex¬ 
tended  into  the  mid-1960's  and  second,  an  examination  of  the 

trend  towards  an  increased  interest  in  developing  more  systematic 
evaluation  approaches  through  applied  research  for  programs 
in  community  development.  In  this  way,  the  focal  point  of 
part  one  is  on  depicting  descriptions  of  early  community 
development  programs,  their  related  evaluation  approaches 
and  techniques  and  its  gradual  movement  from  becoming  less  of 
an  "art"  and  more  of  a  "science"  through  a  growing  interest 
in  developing  more  systematic  evaluation  methodologies  through 
applied  action  research. 

The  second  part  of  the  appraisal  entitled  "Evaluation 
as  a  Science"  will  focus  on  how  the  methodologies  of  social 
action  research  have  been  adopted  and  applied  to  evaluation 
studies  in  community  development  over  the  past  decade.  From 
this  examination  will  emerge  how  a  new  generation  of  community 
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development  practitioners  has  attempted  to  develop  more 
systematic ,  objective  evaluation  methods  through  applied 
social  action  research  for  determining  the  legitimacy  of 
community  development  programs,  to  generally  upgrade  the 
quality  of  their  evaluation  reports  and  more  importantly  to 
demonstrate  accountability  to  support  agencies. 

Both  the  conceptual  groundwork  provided  from  the 
appraisal  of  "Evaluation  As  An  Art"  and  the  practical  pro¬ 
cedures  provided  by  the  review  of  "Evaluation  As  a  Science" 
will  provide  the  guidelines  towards  designing  a  simplified 

evaluation  framework  for  use  by  practitioners  in  the  field  of 
community  development. 

PART  ONE 

COMMUNITY  DEVELOPMENT  EVALUATION  AS  AN  "ART" 

The  need  and  importance  of  program  evaluation  in 
community  development  is  not  a  new  phenomenon.  Historically, 
it  was  recognized  as  early  as  1954  when  the  United  Nations 
sponsored  a  conference  entitled  "Criteria  and  Technical  Assist¬ 
ance  for  Economic  Development".  One  of  the  most  significant 
outgrowths  from  this  conference  was  a  manual  produced  by 
Samuel  Hayes  (1959)  which  offered  a  simple,  straightforward 
guide  to  assess  effects  and  changes  through  quantitative  and 
qualitative  measures  when  a  traditionally  self-sufficient 
community  undergoes  development  through  the  acquisition  of 
modern  technology.  Hayes  would  later  revise  his  manual  in 
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1966  but  apart  from  adding  more  technical  precision  (e.g. 
statistical  sampling  and  use  of  control  groups)  it  remained 
essentially  the  same. 

Hayes  defined  three  possible  types  of  changes  brought 
about  by  community  development  projects;  changes  in  an 
individual's  information  patterns,  skills  and  attitudes; 
changes  in  institutions  and  social  relationships;  and  changes 
in  social  overhead  capital.  A  consideration  of  measuring  both 
short  and  long-term  effects  relative  to  these  three  broad 
categories  of  change  provided  one  of  the  first  broad  evaluation 
bases  for  assessing  changes  in  development.  Hayes  offered 
two  types  of  evaluation:  first,  an  approach  which  assesses 
changes  in  physical  criteria  such  as  harvest  results;  and 
second,  socio-psychological  evaluation  which  focusses  on 
changes  in  such  things  as  attitudes.  It  is  this  second  approach 
that  Hayes  advocated  as  being  more  important  for  community 
development  projects.  Hayes  stressed  that  the  more  quantitative 
are  the  data  to  assess  these  criteria,  the  better  they  are  which 
is  reflective  of  a  more  systematic  and  rigorous  approach  to 
evaluation,  and  which  provides  a  basis  for  future  comparative 
studies  to  be  performed  on  community  development  projects. 

Hayes' manual  provided  the  basis  for  other  evaluations 
of  community  development  projects  in  the  early  1960 's  whose 
focus  was  primarily  on  the  international  rural  and  agrarian 
scene  (e.g.  Indonesia,  Nigeria,  Britain,  India)  and,  as  such 
studies  by  Beers  (1960)  ,  Jain's(1961)  ,  Batten  (1962)  provided 


80 


some  interesting  insights  into  evaluation  approaches  applicable 
to  community  development,  they  will  be  examined. 

Beers'  (1960)  commentary  focused  on  how  evaluation  of 
community  development  projects  in  India  should  be  performed. 

He  contends  that  the  attention  given  to  evaluation  over  the 
past  few  years  gives  it  a  special  status  and  makes  it  an  area 
deserving  more  attention.  He  views  evaluation  as  a  means 
and  is  not  an  end  in  itself  to  assessing  social  action  pro¬ 
jects.  In  this  way.  Beer's  viewed  community  development 
evaluations  as  not  just  an  assessment  of  purposes  but  a 
critique  of  how  stated  objectives  are  sought. 

Beers'  identified  four  rules  by  which  such  an  evaluation 
should  be  conducted.  First,  good  qualitative  criteria  should 
reflect  clear  statements  of  important  immediate,  intermediate 
and  ultimate  goals.  Secondly,  the  means  of  reaching  these 
goals  should  be  clearly  described.  Third,  data  to  be  utilized 
to  achieve  ends  must  be  clearly  established.  And  fourth,  the 
evaluation  should  include  staff,  citizens  and  project  partici¬ 
pants.  He  made  explicit  the  fact  that  such  an  evaluation 
process  should  occur  in  conjunction  with  the  planning  of  the 
entire  project  and  should  occur  again  when  plans  of  action 
can  be  modified.  Moreover,  Beers  criticizes  the  prevalent  use 
of  ex-post-f acto  evaluation  which  often  yields  disappointing 
results  and  fail  to  highlight  important  data  which  occur 
during  the  process  of  change  induced  by  community  development 
intervention.  Events  can  be  evaluated  but  not  just  after  the 
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termination  of  a  project.  In  other  words,  Beers’  emphasis  is 
on  the  evaluation  of  the  community  development  process  and 
not  just  upon  its  outcomes  in  order  to  measure  what  he  terms 
"relative  efficacy  of  gradualism",  an  inherent  quality  of 
developmental  change. 

Jain's  text  Community  Development  and  Panchayati  in 
India  (1961)  provided  a  model  which  attempted  to  make  a  more 
systematic  and  analytic  type  of  evaluation  research  possible 
for  community  development  projects.  This  model  utilized  a 
theoretical  base  for  measuring  outputs  and  identifying  planned, 
unplanned  and  inter-active  effects  that  can  be  utilized  for 
process  and  output  evaluation.  The  model  identified  two 
types  of  growth  or  change  as  the  subject  base  for  evaluation 
in  community  development:  quantitative/aggregative  growth  or 
change  which  is  easily  subjected  to  statistical  analysis;  and 
structural  growth  or  change  which  employs  more  qualitative 
measures.  Jain's  work,  thus,  appears  to  be  one  of  the  first 
attempts  to  evaluate  change  in  terms  of  using  both  quantitative 
and  qualitative  criteria,  the  former  having  the  advantage  of 
being  utilized  in  combination  with  comparative  experimental 
techniques.  He  also  stressed  the  need  for  both  outcome  and 
process  evaluation  procedures  for  community  development. 

The  former  colonial  administrator  T.  R.  Batten 
authored  several  major  texts  and  articles  (e.g.  Training 
for  Community  Development,  A  Critical  Study  of  Method  (1962) 
and  The  Human  Factor  in  Community  Work  (1965))  related  to 
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community  development  and  how  to  assess  its  effectiveness  as 
a  method  to  effect  social  change.  His  emphasis  was  on  the 
use  of  narrative  reports  as  an  analytic  technique.  Batten 
(1962:  1-2)  offered  some  suggestions  for  the  formulation  of 
criteria  in  terms  of  assessing  change  relative  to  measuring 
immediate,  intermediate  and  ultimate  objectives: 

What  criteria  can  we  use  to  measure  success  in  this 
kind  of  work?  Is  it  enough  for  instance,  that  the 
worker  should  achieve  his  immediate  purpose;  .  .  . 

In  fact  this  is  not  enough  .  .  .  for  development  to 

take  place  for  the  worker  really  to  succeed , --the 
changes  he  promotes  must  have  some  lasting  good  effect 
.  .  .  the  more  we  think  of  development  as  a  continuing 

process  each  stage  of  which  has  some  good  or  bad — 
or  the  attitude  of  people  to  the  succeeding  stage, 
the  more  concerned  we  shall  be  about  the  effect  of 
each  stage  of  development  on  the  attitudes  and 
relationships  of  people. 


However,  a  major  drawback  of  Batten's  studies  is  that  they 
appear  to  reflect  a  reluctance  to  perform  more  rigorous, 
systematic  evaluations  of  program  effects  which  Batten  (1962: 
71)  perceives  as  being  antithetical  to  the  principles  of  the 
community  development  process.  .  . 


We  have  never  attempted  any  more  precise  and  any 
systematic  evaluations  partly  because  we  do  not 
think  that  the  results  would  be  sufficiently  meaning¬ 
ful  and  partly  because  we  feel  it  would  be  wrong 
to  try  .  .  .  to  us  measurement  of  results  in  this 

particular  field  is  not  only  difficult  but  also 
repugnant.  Rapport  for  people  is  a  core  concept 
of  CD . 

Such  a  regressive  attitude  towards  evaluation  appears  to  have 
been  shared  by  others  and  accounts  for  why,  apart  from  the 
notable  exceptions  we  have  pointed  out ,  ccmnunity  development  failed 
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to  generate  many  sound  evaluation  studies  in  its  early  history. 

As  community  development  became  adopted  in  North 
America  throughout  the  early  1960's  there  became  a  subsequent 
concern  with  evaluating  its  effectiveness  but  in  a  different 
context.  That  is,  here  community  development  programs  were 
faced  with  more  urban  industrialized  environments  unlike  those 
found  in  rural,  agrarian  villages.  Thus,  community  develop¬ 
ment  and  its  evaluation  process  had  to  be  made  more  relevant 
to  such  settings  while  it  exhibited  more  institutionalized 
constraints  (e,g.  political).  As  the  works  of  James  (1961) 
Clinard  (1963)  Biddle  and  Biddle  (1965)  and  Lee  J.  Car"/  reflect 
significant  insights  into  the  role  of  evaluation  for  community 
development  programs  in  an  urban  context,  they  will  be  examined 
in  the  rest  of  this  section. 

James  (1961:  10)  viewed  community  development  programs 
as  successes  when  they  met  the  criteria  appropriate  to  the 
community  development  movement.  He  offered  the  following  three 
groups  of  criteria,  related  to  the  primary  goal  of  the 
community  development  tradition,  namely,  to  make  a  community 
a  better  place  in  which  to  live:  first,  the  voluntary  establish¬ 
ment  of  a  local  organization  designed  to  meet  the  needs  of  a 
community;  second,  increased  communications  and  interaction 
between  community  members;  third,  the  development  of  procedures 
and  techniques  utilized  by  communities  to  meet  local  needs. 

James  significantly  points  out  that  a  number  of  emerging 
alternatives  provided  by  city  planning  departments  and 
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government  intervention  is  threatening  its  existence.  In 
light  of  these  emerging  political  constraints,  he  contends 
that  it  is  time  to  redefine  community  development  tactics, 
strategies  and  the  ways  of  evaluating  the  effectiveness  of 
its  programs  in  meeting  the  needs  of  the  communities  they  are 
designed  to  serve. 

Marshall  Clinard  (1963)  makes  an  important  distinction 
between  urban  and  rural  community  development  evaluation 
studies.  The  former  he  sees  as  more  difficult  and  concerned 
not  just  with  physical  results.  Clinard  defines  three  areas 
of  concern  in  urban  community  development  evaluation;  program 
content,  the  impact  of  programs  on  the  community,  and  their 
organizational  aspects.  Moreover,  Clinard  offered  a  broader 
base  for  evaluation  going  beyond  the  simple  steps  advocated 
by  Hayes.  He  recommended  that  evaluations  undertake  a  more 
systematic  research  approach  through  utilizing  more  sophisti¬ 
cated  techniques  such  as  participation  rates,  and  surveys  that  can 
yield  more  qualitative  data  and  criteria  measures.  Later, 
Wiledon  (1970)  elaborated  cn  Clinard' s  evaluation  techniques  for  yield¬ 
ing  data,  criteria  and  measures  towards  conducting  evaluation 
research.  They  ranged  from  very  simple  to  more  complicated 
types;  reports  and  record  keeping  of  work  done,  rating  plans, 
score  cards,  how  to  conduct  interviews,  arrange  evaluation 
groups  and  review  case  studies.  Similarly,  Dennis  Poplin 
(1972)  expanded  on  the  techniques  of  community  research,  and 
offered  the  community  development  practitioner  a  basic 
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introduction  on  how  to  conduct  a  social  survey  and  utilize 
data  from  case  histories  and  participation  observation 
techniques . 

The  Biddles  (1965)  also  expanded  upon  the  need  for 
more  systematic  evaluation  in  community  development.  The 
chapter  entitled  "Research  Design"  in  their  major  text  The 
Community  Development  Process  is  really  a  basic  guideline  for 
conducting  social  action  research. 

The  Biddles  (1965)  advocate  an  approach  to  process 
analysis  whereby  an  evaluation  should  emphasize  the  improvement 
of  the  quality  of  the  community  development  process  and  not 
simply  the  assessment  of  changes  occurring  within  the  indivi¬ 
dual.  Their  emphasis  is  on  the  use  of  narrative  reports. 
Records  are  kept  of  the  results  of  meetings  and  accounts  of 
action  taken.  Such  records  and  non- judgemental  accounts,  they 
argue,  should  be  reviewed  periodically  by  community  development 
workers,  researchers  and  citizens.  These  narrative  reports 
include  data  and  some  statistics  from  which  can  be  derived 
general  hypotheses  and  criteria  for  determining  what  constitu¬ 
tes  improvements  in  people/individuals  undergoing  change 
processes:  The  Biddles  stated  (1965:  144)  that  an  evaluation 
should  consider 

.  .  ,  qualitative  and  quantitative  treatment  of 

describable  items  of  experience  in  the  flow  of 
process.  These  should  finally  lead  to  conclusions 
about  man  in  development  which  then  become  new 
general  hypotheses  for  further  testing. 
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The  Biddles’  emphasis  upon  moving  from  narrative  reports  to 
taxonomies  and  hypothesis  testing  significantly  reflects 
an  interest  in  advancing  more  systematic  evaluation  techniques 
through  applied  action  research. 

Lee  J.  Cary  edited  a  text  (1970)  ,  Community  Development 
As  a  Process  which  contains  some  chapters  on  evaluation. 
Although  the  author's  works  in  Cary's  text  remain  at  the 
conceptual  level  offer  no  specific  criteria  and  to  evalu¬ 
ate  change,  the  works  of  Sanders,  Sutton  and  Schler  do  offer  significant 
insight  and  leading  statements  and  principles  towards  conduct¬ 
ing  systematic  evaluations  in  community  development.  For 
example , Irwin  Sanders  (1970)  stressed  the  need  for  applied 
theory  in  community  development  and  argued  that  the  "applica¬ 
tion  of  social  service  methods  can  be  used  to  try  and  clarify 
the  process  through  which  community  changes  are  achieved". 

Also,  Sutton  (1970:  80)  urged  that  evaluations  in  community 
development.  .  . 

should  demonstrate  the  consistency  between  action 
episodes  and  societal  values  . 

These  action  episodes  are  a  way  of  subjectating  the  community 
development  process  to  empirical  research.  His  breakdown  of 
action  episodes  is  in  terms  of  group  cohesion,  social  capital 
"know  how"  and  the  use  of  financial  resources.  And  the  relat¬ 
ionship  of  these  four  factors  to  the  success  of  a  community 
development  program  suggest  areas  of  evaluating  change.  Finally, 
Daniel  Schler  (1970:  142)  emphasizes  process  evaluations, 
suggestions  for  criteria,  and  the  importance  of  considering 
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wide  effects  when  assessing  long  range  goals  of  community 

development  as, in  his  words: 

Those  who  study  the  effects  of  community  development 
programs  .  .  .  [are]  advised  to  include  evaluation 
instruments  to  determine  to  what  degree  prospective 
goal  setting,  self  help,  joint  enterprise  endeavors, 
democratic  human  relationships  and  rational-concerted 
decision-making  and  action  have  become  institutional¬ 
ized  into  the  over-all  structure  of  a  community  system 
.  .  .  to  consider  the  tendencies  for  behavioral 
and  social  change  that  the  community  development  process 
implies.  .  .  .  Upon  closer  observation  it  may  be  dis¬ 
covered  that  the  residual  results  are  more  important 
for  human  survival  in  the  long  run  than  the  immediate 
goals . 

The  provision  of  the  conceptual  basis  for  conducting 
more  systematic  evaluations  in  community  development  through 
applied  research  was  a  trend  that  continued  in  the  early  1970's. 
For  example,  Hobgood  (1972)  noted  that  the  tendency  in  community 
development  work  was  to  move  directly  from  theoretical  to 
concrete  recommendations  without  an  investigation  of  the 
particular  situation  which  could  be  systematically  investigated 
through  various  types  of  applied  research  (e.g.  descriptive 
and  economic,  problem  orientated,  operations  and  professiona- 
ism) .  Hobgood  contended  that  evaluations  should  be  directed 
at  assessing  socio-economic  and  personal  effects  resulting 
from  community  development  programs  in  terms  of  both  material, 
physical  criteria  and  qualitative  criteria  such  as  changes  in 
people's  values ,  attitudes,  perceptions  or  prejudices.  Hobgood 
saw  evaluation  research  as  being  central  to  all  other  forms 
of  research,  and  it  should  become  more  widely  used  in  community 
development  as  it  provided  the  basis  of  growth  and  development 
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that  prevented  any  program  or  organization  from  degenerating, 
stagnating  or  repeating  past  mistakes. 

Similarly,  Epstein,  Tripodi  and  Fellin  (1973)  have 
noted  the  recent  rising  demand  for  community  development 
evaluations  as  community  development  administrators  have  come 
under  increased  pressure  from  funding  sources,  professional 
pressure  groups  and  clientele  to  provide  systematic  evaluations 
of  their  programs.  The  broadening  of  the  term  evaluation  as 
a  management  technique  for  the  administration  of  community 
development  programs  is  particularly  novel  to  these  three 
researchers.  They  recognize  that  through  evaluations,  community 
development  administrators  can  provide  a  framework  for  decision 
making  about  programs  based  on  available  and  objective  infor¬ 
mation.  This,  in  turn,  can  be  used  to  provide  more  efficient 
and  effective  community  development  programs.  Furthermore, 
as  did  Hobgood,  they  broaden  the  term  "evaluation  research" 
to  encompass  other  than  traditional  research  activities  such 
as  pure,  basic,  or  operational  and  encourage  community  develop¬ 
ment  administrators  to  select  from  the  available  research 
techniques  those  which  can  be  most  appropriately  utilized  for 
evaluating  social  action  community  development  programs. 

In  summary,  the  appraisal  of  literature  to  this  point, 
illustrates  that  there  exists  a  minimal  amount  of  evaluation 
literature  from  the  early  history  (early  1960's)  of  community 
development  and  those  existing  evaluation  studies  were  based 
on  unsystematic  approaches  and  dealt  primarily  with  theoretical 
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concepts  and  reflected  an  interest  mainly  in  problem  defini¬ 
tion  rather  than  in  ways  and  means  of  applying  evaluation  to 
community  development  programs.  Nevertheless,  those  significant 
leading  statements  on  the  techniques  and  basis  for  formulating 
a  framework  related  to  evaluations  in  community  development 
can  be  summarized: 

1.  The  effects  of  change  (e.g.  upon  the  individual 
institutions,  and  overhead  capital)  should  be  assessed  in 
terms  of  both  short-term  and  more  importantly  long-term 
objectives . 

2.  The  evaluation  process  should  be  conducted  in 
accordance  with  the  way  in  which  a  program  was  planned. 

3.  The  emphasis  should  be  upon  assessing  ongoing 
effects  of  change  initiated  by  the  community  development 
process  and  not  just  upon  its  outcome  in  light  of  revealing 
partial  results  whose  significance  must  be  considered  in  order 
to  more  properly  assess  the  program's  success. 

4.  Both  quantitative  and  qualitative  criteria  and 
measures  should  be  developed  towards  assessing  program  object¬ 
ives  . 

Also,  it  has  become  apparent  from  this  appraisal  that 
the  mid  1960's  witnessed  the  emergence  of  a  new  generation  of 
community  development  practitioners  which  recognized  the  need 
for  more  evaluation  studies  in  community  development  and  noted 
that  they  should  reflect  more  systematic,  objective  evaluation 
approaches  for  determining  the  legitimacy  of  community  develop¬ 
ment  programs,  to  generally  upgrade  the  quality  of  community 
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development  evaluation  reports,  to  demonstrate  accountability 
to  support  agencies.  However,  these  efforts  remained  primarily 
at  the  theoretical  basis  and  remained  primarily  as  anectodal 
case  studies  (MacKeracher,  Davie,  Patterson,  1976).  In  this 
way  evaluation  in  community  development  could  be  perceived 
of  as  an  "art"  (Voth,  1975,  and  Lafleur,  1977)  as  such  studies 
reflected  little  attempt  to  go  beyond  the  conceptual  level 
and  provide  for  systematic,  rigorous  evaluation  studies 
through  the  provision  of  frameworks  for  practical  use.  How¬ 
ever,  an  exception  to  this  generalization  can  be  found  in  a 
study  conducted  by  Lovell  and  Riches  (1967-1968)  who  develop¬ 
ed  a  framework  which  integrated  both  quantitative  and  quali¬ 
tative  criteria  and  measures  for  systematically  evaluating 
attitude  changes  in  the  individual  which  occurs  from  his  in¬ 
volvement  in  a  community  development  program.  They  hoped  that 
such  a  framework  would  aid  community  development  workers  to 
evaluate  more  systematically  the  progress  of  their  work  and 
the  effectiveness  of  their  methods. 

Lovell  and  Riches  defined  the  two  primary  objectives 
of  community  development  as;  the  betterment  of  environments 
and  the  betterment  of  people.  The  former  can  be  measured 
relatively  easily  in  quantitative  terms  (e.g.  facilities). 

The  latter  change  however  is  more  difficult  to  measure  as  it 
requires  the  measurement  of  qualitative  items  such  as  changes 
in  attitudes.  The  two  researchers  went  on  to  identify  a  group 


91 


of  nine  major  attitudes  which  must  change  if  a  program  is 
to  be  perceived  as  a  success:  apathy,  feelings  of  inadequacy, 
self  centeredness ,  ignorance,  prejudice,  mistrust,  gullibility, 
impetuosity  and  instability.  Next,  Lovell  and  Riches  formu¬ 
lated  17  measurable  criteria  with  which  to  assess  whether  an 
increase  or  decrease  has  occurred  in  some  or  all  of  these 
characteristics.  Lovell  and  Riches  then  compiled  a  list  of 
seventy-six  indicators  of  these  criteria  which  include  both 
quantitative  and  qualitative  items.  They  continued  their 
analysis  by  attempting  to  illustrate  pictorially  through 
charts  and  graphs  the  differential  applicability  of  the 
various  criteria  to  the  nine  major  attitudes  requiring  change, 
thereby  providing  an  overall  picture  of  change  in  depth.  The 
charts  and  graphs  provided  by  Lovell  and  Riches  provide  the 
first  significant  attempt  to  provide  the  evaluator  with  a 
visible  picture  with  which  to  hypothetically  assess  change  in 
a  more  systematic  and  rigorous  way. 

In  short,  Lovell  and  Riches  study  appears  to  be  a 

significant  "stepping  stone"  bridging  the  gap  between  community 

development  being  perceived  as  a  "science"  rather  than  as  an 

"art".  That  is,  the  framework  they  developed  reflected  one 

of  the  first  significant  attempts  to  go  beyond  the  conceptual 

level  and  to  actually  provide  more  systematic  evaluation 

studies  (e.g.  the  development  of  criteria  and  measures)  in 
$ 

community  development,  a  trend  which  would  continue  throughout 
the  next  decade  through  an  application  of  social  action  method¬ 
ologies  . 
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PART  TWO 

COMMUNITY  DEVELOPMENT  EVALUATION  AS  A  "SCIENCE" 

Although,  as  Burton  (1975:  45)  pointed  out,  there 
still  remains  a  lack  of  sound  evaluation  studies  in  community 
development  it  appears  from  the  increasing  body  of  literature 
that  the  interest  in  evaluation  studies  generated  in  the  1960's 
has  continued  to  grow  in  the  field  over  the  past  decade  towards 
providing  for  more  effective  evaluation  studies.  This  trend 
has  been  particularly  prompted  by  the  fact  that  today  particu¬ 
larly  in  the  face  of  growing  demands  for  accountability  from 
funding  sources,  there  has  become  as  Burton  noted  (1978:  45)  a 
growing  insistence  on  and  call  for  more  and  better  evaluation. 
Consequently,  community  development  practitioners  are  devoting 
more  time  and  resources  towards  providing  more  systematic 
program  evaluations.  Moreover,  such  systematic  methodologies 
are  based  on  many  of  the  attributes  of  social  action  research 
which  has  been  defined  as  having  an:  emphasis  on  the  introduc¬ 
tion  and  observation  of  planned  change;  problem  centered, 
emphasizing  some  improvement  on  some  socially  desirable  goal; 
utilizes  partial  use  of  empirical  science;  emphasizing  a 
systems  approach;  is  most  effective  when  applied  in  complex, 
rapidly  changing  systems;  concerned  with  variables  which  appear 
to  make  some  difference  in  the  performance  of  the  system  (e.g. 
utilization  of  human  resources,  control  and  leadership 
collaboration  and  conflict) ;  focuses  on  the  quality  and  nature 


93 


of  relationships  with  the  client  group  or  organization  which 
is  the  target  of  the  planned  change;  and  the  relationship 
between  the  action-researcher  and  evaluator  presents  unique 
role  requirements.  (Carter  and  Wharf,  1973:  31).  Moreover 
unlike  other  forms  of  traditional  research  such  as  pure,  basic 
or  operational,  action  research  appears  to  appropriately  handle 
the  value  laden  nature  of  both  evaluation  and  the  community 
development  process.  Consequently,  as  pointed  out  by  Burton 
(1978:  45)  research  as  part  of  the  evaluation  process  has  been 
made  more  meaningful  when  seen  in  its  proper  place  in  that  process. 

As  it  is  an  investigation  of  evaluations  in  community 
development  as  applied  action  research,  it  then  follows  that 
the  material  in  this  chapter  will  be  organized  to  cover  the 
four  evaluation  levels--input ,  process,  product,  context — and 
corollary  evaluation  activities  (pref ormative ,  formative  and 
summative) . 

The  purpose  of  this  organizational  basis  is  to  examine 
the  role  of  evaluation  based  on  applied  social  action  research 
and  the  extent  to  which  it  has  contributed  towards  more 
systematic  evaluations  in  community  development  and  hence, 
moving  it  in  the  direction  of  being  perceived  of  as  a  "science" 
and  not  of  an  "art". 

INPUT  EVALUATION  LEVEL 

The  Setting  of  Objectives 


Voth  noted  (1975:  150)  that  one  of  the  major  conceptual 
problems  inherent  within  community  development  programs  is  that 
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goals  are  often  ambiguous  vague,  abstract  and  complex  and 
thus, very  difficult  to  state  in  operational  terms.  However, 
it  appears  that  more  precise  objective  guides  and  sound  indi¬ 
cations  for  evaluating  community  development  programs  have 
been  acquired  through  applying  the  techniques  of  social  action 
research;  the  conceptualization  of  objectives  performed  in 
accordance  with  the  theoretical  premises  of  the  overall  program 
plan; and  in  addition  with  the  preformative  evaluation  activity 
defined  as  needs  assessment  which  provides  an  informational 
base  or  input  into  the  setting  of  objectives. 

Beal,  Coward,  and  Brooks  (1971  :  57)  made  one  of  the  first 
conscious  efforts  to  reduce  the  intangible  nature  of  community 
development  goals  which  are  often  reduced  to  the  level  of 
various  program  actions  with  which  the  organization  is  familiar 
and  which  it  is  willing  to  implement.  They  specify  development 
objectives  according  to  a  means — end  schema  conceptualizing 
the  range  which  they  have  established  in  accordance  with  the 
theoretical  premises  of  the  community  development  process. 

Such  a  schema  helps  to  discover  the  important  linking  or  inter¬ 
vening  variables  occurring  in  the  change  process  that  provide 
guidelines  on  what  to  measure  and  howto  measure  it. 

Specifically,  the  model  developed  by  the  authors 
illustrates  the  change  sequence  between  socio-psychological 
and  structural  variables  and  indicates  the  assumed  direction 
of  the  causal  relationships.  It  assumes  that  institutions  and 
social  conditions  have  effects  upon  an  individuals'  behaviour 
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and  attitudes  which  either  enhance  or  retard  institutional 

development.  Such  a  causal  relationship  depicted  from  the 

theoretical  premises  of  community  development  is  noted  later 

by  Schabel  and  Sinani  (1975:  113)  who  saw  that.  .  . 

unless  community  development  efforts  result  in 
such  things  as  a  rising  level  of  housing,  recreation, 
access  to  goods  and  services  necessary  for  psycholo¬ 
gical  and  physiological  welfare  effective  development 
has  not  occurred. 

Finally,  after  setting  the  overall  program  goals  Beal, 
Coward  and  Brooks  (1971:  48)  suggest  conceptualizing  the 
range  of  development  objectives  and  development  measures  to 
help  clarify  when  goals  are  not  being  met.  In  order  to  achieve 
this  a  taxonomy  of  a  range  of  development  goals  should  be 
developed  so  that.  .  . 

general  level  concepts  in  the  taxonomy  could  be 
operationalized  to  lower  sub-components  which  provide 
clearer  direction  for  program  activities  and  the 
measurement  of  results. 

Such  a  taxonomy  is  reflective  of  Suchman's  ordering  of  object¬ 
ives  into  immediate,  intermediate  and  ultimate  categories  in 
order  to  guide  the  researcher  in  conceptualizing  objectives 
and  developing  precise  and  appropriate  measures  to  evaluate 
their  effects.  This  taxonomy  has  been  applied  by  Bennett  (1975) 
(see  Chapter  2,  pp.  5-6)  towards  evaluating  extension  programs 
and  will  be  similarly  used  in  the  framework  developed  in  this 
thesis . 


A  significant  development  is  that  there  has  evolved  a 
clear  recognition  that  there  are  two  separate  sets  of 
community  development  goals:  task/concrete  objectives  and 
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process  objectives  which  according  to  Cebotarev  and  Brown 
(1972:  45)  are  merely  two  different  change  strategies  working 
towards  the  same  ultimate  goal  of  improving  the  quality  of 
life  for  members  in  a  community  engaged  in  a  community  develop¬ 
ment  program. 

Task  orientated  obj ectives  refer  to  development  sequences 
directed  at  changes  in  the  social/physical  environment  (e.g. 
institutions,  services  or  facilities  which  a  community  needs 
and  wants)  and  emphasize  change  in  quantitative  terms.  In 
contrast  process  objectives  are  according  to  Voth  (1975:  43)  .  .  . 

not  content,  i.e.,  facilities  or  services  of  some 
kind,  but  initiation  of  a  process  in  which  all 
people  of  a  community  are  involved,  through  their 
representatives,  in  identifying  (taking  action  in 
respect  to  their  own  problems.  .  .  .  The  objective 

is  less  that  of  some  specific  reform  that  is 
development  of  community  integration;  capacity  to 
function  as  a  unit  in  respect  to  common  problems. 

In  other  words,  process  objectives  reflect  the  ability  of 
the  community  to  solve  its  problems  and  make  collective 
decisions.  In  this  way  the  process  objectives  reflect  qualita¬ 
tive  changes  which  community  development  programs  try  to  effect 
and  as  they  embrace  the  same  theoretical  principles  of  the 
community  development  process — a  process  of  purposive  social 
change  evaluating  community  citizens  participation  in  the  decision¬ 
making  process  of  defining  and  meeting  their  own  felt  needs-- 
it  is  not  surprising  that  the  setting  and  evaluation  of  qualita¬ 
tive  changes  related  to  process  objectives  is  the  focal  point 
of  the  majority  of  community  development  evaluations.  Thus,  as 
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MacKeracher,  Davie  and  Patterson  (1976:  6)  point  out  an  evalu¬ 
ator  needs  to  decide  which  he  wants  to  examine,  either  the 
process  objectives,  task  objectives,  or  both,  prior  to  design¬ 
ing  an  evaluation  system  because  the  failure  to  distinguish 
between  the  approaches  lead  to  a  failure  to  develop  criteria 
and  measures  appropriate  to  each.  For  example,  the  miscon- 
ceptualization  of  the  program  as  task  orientated  rather  than 
process  orientated  was  responsible  for  the  failure  of  an  evalu¬ 
ation  performed  on  community  development  programs  in  India  by 
Gavin  Kavinaratne  (1976) . 

In  short,  the  conceptualization  and  measurement  of 
program  objectives  has  been  performed  with  more  ease  in 
community  development  when  done  in  accordance  with  the  theoret¬ 
ical  premises  of  the  overall  program  plan.  Moreover,  the 
theoretical  premises  of  a  community  development  program  indicate 
that  there  are  two  separate  sets  of  objectives,  namely,  task 
and  process  for  which  the  community  development  practitioner 
must  define  both  quantitative  and  qualitative  criteria  and 
measures  in  order  to  assess  their  effects  upon  the  consumers 
of  a  community  development  program. 

Community  Needs  Assessment 

It  would  seem  useful  for  the  purposes  of  this  thesis  to 
broaden  the  concept  of  needs  assessment  to  that  of  community 
assessment,  which  according  to  Mitchell  (1977:  33)  is  an 
analysis  of  community  strengths,  weaknesses  and  the  determination 
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of  priorities  with  consideration  being  given  to  already  exist¬ 
ing  resources  and  provides  an  important  informational  base 
which  aids  in  the  conceptualization  and  setting  of  objectives 
most  relevant  to  the  program  under  evaluation.  A  familiari¬ 
zation  with  the  most  widely  used  techniques  for  conducting 
community  needs  assessment  (e.g.  citizen  survey)  has  been  per¬ 
formed  in  chapter  2.  It  is  an  investigation  of  their  applica¬ 
tion  in  community  development  which  will  be  the  focus  of  this 
section . 

Blake,  Kalb,  and  Ryan  (1977)  noted  that  citizen  surveys 
have  played  an  important  role  in  conducting  community  needs 
assessments,  several  of  which  will  now  be  examined. 

Nix's  (1971)  Community  Reconnaissance  Method  was  one 
of  the  first  efficient,  quick  techniques  for  assessing  relevant 
aspects  of  the  community's  social  structure  and  process  through 
the  use  of  leader  respondants  as  respondents.  According  to 
Nix  (1971:  69)  the  purpose  of  the  survey  is  to  help  local 
citizens,  leaders  and  groups  to  .  .  . 

1.  identify  the  felt  needs:  and  problems  of  the 
community . 

2.  rank  in  priority  the  need;  problem  areas  to  be 
dealt  with. 

3.  organize  or  mobilize  to  deal  with  chosen  needs: 
problems . 

4.  study  the  identified  needs:  determine  specific  goals. 

5.  develop  a  plan  of  action  to  accomplish  locally- 
determined  goals. 

6.  find  resources  to  accomplish  goals. 

7.  act  to  accomplish  goals. 

8.  evaluate  accomplishments. 


. 
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Nix  concludes  that  through  the  use  of  his  method,  the  assess¬ 
ment  of  the  community's  social  structure  and  processes  can 
provide  an  important  informational  base  towards  more  effective 
planning  of  programs  designed  to  effect  change  in  a  communitv. 

Cohen,  Sills  and  Schwebel  (1977)  advocate  a  two  stage 
process  for  surveying  community  needs  that  is  based  on  an  open- 
ended  survey  and  close-ended  questionnaire.  By  using  this 
two  stage  interview  process  a  wider  range  of  people's  problems 
can  be  uncovered  and  can  help  to  determine  the  prevalence  and 
severity  of  such  problems  in  the  service  area.  Such  a  procedure 
helps  to  reduce  much  of  the  bias  inherent  within  questionnaire 
writing  and  provides  the  researcher  with  a  more  realistic 
picture  of  community  needs  based  on  more  input  for  the  people 
in  the  community. 

Dillman  (1977:  33-37)  noted  that  due  to  constraints 
of  expenses  the  researcher  is  forced  to  rely  on  "independent 
surveys"  which  he  feels  are  inadequate.  Such  surveys  are 
undertaken  independently  from  the  actual  policy  setting  process 
and  contain  no  clear  definition  founded  in  policy  issues  but 
rather  those  definitions  produced  from  "arm-chair  deliberations" 
whereby  questions  from  previous  surveys  are  revitalized  and 
in  this  way  general  rather  than  specific  attitudes  are  sought. 
Dillman 's  alternative  is  the  "synchronized  survey"  which 
is  conducted  through  mail  and  telephone  data  collection 
methods.  The  "synchronized  survey"  demands  that  the  survey 
researcher  spend  most  time  interacting  with  all  decision-makers 
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both  the  policy  makers  and  citizens  and  thereby  doing  a  much 
better  job  "of  measuring  people's  preferences  and  bringing 
them  to  bear  on  policy  decisions"  than  done  in  the  past  when 
only  independent  surveys  were  conducted. 

In  short ,  it  appears  that  significant  attempts  have 
been  made  to  make  community  needs  assessments  a  more  co-opera¬ 
tive  effort  between  human  service  management  and  those  it  is 
designed  to  serve.  In  this  way,  it  appears  that  such  approaches 
are  acquiring  greater  community  development  orientation  whereby 
citizens  actively  participate  in  the  decision-making  processes 
of  defining  their  own  felt  needs  and  informational  base,  thereby 
assisting  in  providing  input  for  the  conceptualization  of  object¬ 
ives  . 

In  summary,  the  input  evaluation  level  in  conducting 
evaluations  of  community  development  programs  has  increasingly 
applied  the  principles  of  social  action  research;  the  setting 
of  objectives  in  accordance  with  the  theoretical  premises  of 
the  overall  program  plan;  and  the  utilization  of  input  or 
information  received  from  a  community  needs  assessment  towards 
conceptualizing  program  objectives. 

PROCESS  EVALUATION  LEVEL 

Specification  of  Measures 

In  short,  the  preceeding  section  indicates  that  the 
responsibility  of  the  community  development  practitioner  is  on 
evaluating  the  changes  which  a  community  development  program 
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attempts  to  effect  in  the  community  in  both  quantitative  (task 
objectives)  and  qualitative  terms  (process  objectives) . 

Defining  measures  for  task  objectives  is  fairly  easy 
to  establish  as  they  deal  with  physical  things  which  can  be 
seen  and  touched  (e.g.  facilities)  and  can  be  easily  assessed 
through  quantitative  measures  (e.g.  numbers  existing  in  a 
community) .  However,  defining  measures  to  assess  process 
objectives  referring  to  socio-psychological  development  such  as 
changes  in  institutions  (e.g.  local  decision-making  processes) 
or  changes  in  the  individual  (e.g.  behavioural  changes)  are 
more  difficult  to  establish  as  they  deal  with  qualitative 
variables  that  one  cannot  touch  or  easily  observe.  But  as 
they  are  the  primary  focus  of  evaluation  studies  in  community 
development  then  every  effort  must  be  made  to  develop  both 
quantitative  and  qualitative  criteria  and  measures  which  as 
Reyburn  (1975:  56)  contends  will  produce  more  professionally 
competent  evaluations  in  community  development. 

The  measuring  of  behavioural  changes  has  been  the 
objective  of  many  community  development  evaluations  and  includes 
such  things  as  measuring  changes  in  knowledge,  skill ,  aspirations. 
The  study  by  Lovell  and  Riches  (1967-1968)  reflected  a  significant 
attempt  at  developing  both  quantitative  and  qualitative 
measures  with  which  to  measure  attitude  changes  (see  pp.  14-15) . 
Also,  as  Reyburn  (1975:  56)  points  out,  the  measurement  of 
changes  in  knowledge  and  skills  has  now  been  performed  exten¬ 
sively  and  with  relative  ease  in  community  development. 
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Reyburn  suggests  that  aspirational  change  deserves  more 
attention  and  could  be  measured  through  simple  pre-  post-tests 
that  could  indicate  if  the  original  career  aspirations  have 
been  brought  closer  to  those  perceived  by  the  program  recipients. 
Reyburn  concludes  that  as  practice  changes  (e.g.  changes  in 
local  decision-making  processes)  affect  the  whole  society, 
then  evaluation  studies  here  are  another  important  focus  for 
community  development.  It  appears  that  Reyburn  is  suggesting 
a  broader  evaluation  base  for  community  development;  in  other 
words  to  not  confine  itself  to  concentrating  on  assessing  only 
behavioural  changes. 

Voth,  (1975:  151)  indicates  that  what  needs  to  be 
achieved  is  to  state  objectives  more  clearly  with  respect  to 
the  individual,  family,  group,  neighbourhood  or  community  in  order 
to  place  such  objectives  into  more  operational  terms  for 
which  can  be  developed  more  precise  and  empirical  criteria  and 
measures.  A  notable  example  of  how  this  has  been  achieved 
lies  in  a  study  conducted  by  Smith  (1977:  31)  who  contended 
that  community  development  researchers  need  to  borrow  and 
modify  the  performance  criteria  defined  in  other  areas,  such 
as  education,  in  order  to  develop  performance  criteria  appropriate 
specifically  to  community  development.  In  his  evaluation 
scheme  Smith  lays  out  eleven  attributes  from  which  ideally 
researchers  could  develop  criteria  and  measures  of  desirable 
field  performance  in  community  development.  Smith  contends 
that  criteria  of  performance  should  be  developed  through  a  three 
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stage  cyclical  process  where  at  each  phase  measurement 
criteria  are  refined.  According  to  Smith  (1977:  33)  phase 
one  should  develop  a  definition  of  effective  community  develop¬ 
ment  performance.  Phase  two  then  defines  criteria  and  instru¬ 
ments  based  on  this  definition.  And  finally,  phase  three 
should  validate  instruments  utilizing  research  designs  and 
predictable  valid  techniques  such  as  interviews ,  rating  scales, 
performance  tests  and  observation  schedules.  Analysis  of  data 
can  also  include  more  sophisticated  techniques  such  as  Critical 
Incidents  Analysis  or  Retranslation  of  Expectation.  In  reply 
to  the  following  question,  laid  out  by  Smith  (1977:  38).  .  . 

1.  what  aspects  of  community  development  are  reliable? 

2.  how  can  community  development  performance  be 
reliably  observed? 

3.  how  can  reliable  bias-free  ratings  or  judgements 
of  performance  be  obtained? 

4.  what  discrete  performance  indicators  are  likely  to 
be  most  relevant,  empirically  related  to  global 
assessment  of  adequate  performance. 

can  be  derived  partial  answers  that  can  aid  in  the  development 

of  more  refined  measurement  procedures  which  in  turn  can 

provide  more  sophisticated  ways  of  answering  questions 

Smith  concludes  that  through  the  use  of  his  framework  can  be 

developed  a  more  empirical  approach  for  performance  evaluation 

in  community  development.  Smith  hopes  that  it  will  provide  a 

starting  point  from  which  the  growing  number  of  community 

developers  can  more  accurately  evaluate  their  impact,  and  define 

and  measure  levels  of  field  performance  in  a  more  rigorous  and 

systematic  way. 
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In  short,  the  preceeding  section  has  illustrated  that 
both  quantitative  and  qualitative  criteria  and  measures  directed 
at  assessing  task  and  process  objectives  must  be  developed  by 
the  community  development  practitioner.  Moreover,  when  dealing 
with  qualitative  changes  (e.g.  process  objectives)  and  the 
development  of  subsequent  criteria  and  measures,  there  is  a 
great  deal  of  demand  placed  upon  the  evaluator  to  apply 
judgement,  scientific  precision  and  subsequently,  to  select 
the  most  appropriate  research  design  to  study  and  evaluate  the 
effects  of  both  quantitative  and  especially  qualitative  change. 

Research  Designs 

The  trend  in  community  development  has  been  towards 
adopting  descriptive  designs  which  appear  to  be  the  most 
feasible  and  advantageous  towards  studying  quantitative  and 
especially  qualitative  changes. 

Through  the  use  of  descriptive  designs,  more  than  one 
stage  of  the  causal  process  is  incorporated  into  the  analysis. 
Data  are  collected  on  what  actually  occurred  during  the  process 
of  various  program  stages;  how  the  program  was  conducted;  who 
participated  in  particular  activities;  and  the  quality  of  their 
participation  based  on  psychological  behaviour  changes  in  such 
things  as  knowledge,  skills,  aspriations  and  attitudes  which 
have  been  the  target  of  many  community  development  evaluators. 

In  this  way  research  can  be  made  more  meaningful  when  seen  in 
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its  proper  place  in  the  evaluation  process,  as  it  enables  the 
community  development  practitioner  to  more  holistically 
examine,  measure  and  understand  the  effects  of  qualitative 
changes  brought  about  by  community  development  programs. 

MacKeracher,  Davie  and  Patterson  (1976:  5)  have  noted 
the  lack  of  dependable  and  descriptive  methods  for  making 
evaluative  judgements  about  the  effects  of  changes  brought 
about  by  community  development  programs.  The  authors  have 
provided  a  dependable  and  descriptive  method  for  data  collect¬ 
ion  and  making  judgements  known  as  the  SHAPES  SYSTEM  (Shared 
Process  Evaluation  System) .  It  is  a  useful  descriptive  evalu¬ 
ation  system  that  involves  the  collection  of  a  material  in  a 
way  that  more  clearly  documents,  understands,  monitors,  and 
evaluates  the  process  undertaken  in  community  development 
programs . 

SHAPES  incorporates  the  setting  of  goals  by  both  the 
community  agents  and  the  community  participants  in  order  to 
reveal  patterns  of  shared  change.  In  this  way  it  remains 
sensitive  to  the  people's  needs,  as  the  success  of  a  community 
development  program  ultimately  depends  on  whether  the  people 
themselves  benefitted  from  the  process.  One  of  the  most  bene¬ 
ficial  aspects  of  the  SHAPES  is  that  it  provides  visual 
pictures  of  the  process  in  the  form  of  a  set  of  matrices. 

This  is  helpful  in  assisting  agents  and  participants  to  see 
the  pattern  of  what  actually  occurred. 


106 


Voth  (1975:  156)  noted  the  great  need  for  dynamic 
causal  models  in  community  development  which  would  guide  the 
evaluator  in  formulating  his  descriptive  research  design. 
Moreover ,  he  alluded  to  the  fact  that  such  a  model  laying  out 
in  their  respective  order  the  cause  and  effect  variables  could 
be  achieved  in  accordance  with  the  theoretical  premises  of  the 
program.  Later,  Lafleur  (1977:  98)  advocated  the  use  of  the 
"Theory  Based  Evaluation  Approach"  which  would  serve  as  an 
overriding  paradigm  for  the  researcher  in  implementing  a 
research  design  most  appropriate  towards  evaluating  community 
development  programs.  In  this  approach  according  to  Alkin  and 

Fitz-Gibbon  (1975:  11).  .  . 

when  a  program  is  designed  to  implement  a  model, 
a  theory  or  a  philosophy,  the  evaluator  must  select 
his  variables  in  conformity  with  that  model  theory 
or  philosophy.  He  may  or  may  not  measure  the  outcomes 
predicted  by  the  theory.  This  type  of  evaluation 
has  been  called  "theory-based"  evaluation. 

In  this  way  the  theory  points  out  a  causal  relation¬ 
ship  between  a  process  A  and  an  outcome  B.  In  other  words, 

A  leads  to  B  or  causes  B.  When  a  theory-based  evaluation 
is  planned,  the  variables  chosen  for  study  are  those  which 
a  theory  indicates  are  crucial  in  producing  the  desired  program 
outcomes.  The  theory  chosen  is  defined  and  the  degree  of  its 
opc rat ion al i z at ion  within  the  program  is  documented.  lheory 
based  evaluations  would  resemble  those  traditionally  used  to 
measure  the  degree  of  program  implementation  through  tools  such 
as  questionnaires,  records,  interviews  and  observations. 
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According  to  Alkin  and  Fitz-Gibbon  (1975:  3)  Theory  Based 

Evaluations  may  be  used  when  .  .  . 

the  evaluation  of  -process  must  be  attempted  and 
the  program  is  aiming  at  distant  or  intangible 
outcomes  when  the  program  evaluated  is  itself  based 
upon  a  theory,  model  or  philosophy. 

The  above  quotation  explicitly  emphasizes  an  examination  of  the 
process  whereby  change  occurs  rather  than  an  emphasis  on  out¬ 
come.  The  importance  of  this  in  the  evaluation  of  community 
development  programs  has  been  well  documented  by  Cohen  (1976) . 

Cohen  (1976:  19)  noted  the  importance  of  an  ongoing 
examination  of  process  which  reveals  significant,  unantici¬ 
pated  side  effects  which  must  be  accounted  for  in  determining 
the  real  success  or  failure  of  a  community  development  program. 
Such  important  partial  results  can  go  unnoticed  if  attention 
is  primarily  focussed  on  the  program's  outcome.  For  example, 
Cohen  cited  an  evaluation  study  conducted  on  community  develop¬ 
ment  programs  in  Ohio  that  attempted  to  increase  citizen  parti¬ 
cipation.  By  its  outcome  the  program  appeared  to  fail  in 
reaching  its  goals;  but  a  further  analysis  of  process  showed 
that  success  had  totally  been  achieved  as  a  number  of  factors 
from  the  implementation  really  did  have  a  positive  impact  on 
the  community. 

One  of  the  most  recent  studies  to  illustrate  the 
importance  of  process  evaluations  in  light  of  revealing  positive 
partial  results  has  been  conducted  by  Daley  and  Winter  (1978) 
who  designed  a  community  evaluation  model  to  study  the  impact 
on  communities  from  a  nutrition  program  implemented  by  Peace 
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Corps  volunteers.  They  stated  that  Peace  Corps  community 
development  programs  have  not  lacked  for  tangible  results  in 
the  form  of  physical  improvement  projects  which  yield  the 
basic  measures  of  impacts.  However,  they  contend  that  no 
definition  of  success  should  be  restricted  to  such  material 
products, as  the  evaluation  of  Peace  Corps  only  in  terms  of  the 
formal  pre-stated  objectives  would  miss  the  point  of  their 
important  community  development  process  goals  such  as  democratic 
participation  (i.e.  objectives  to  increase  the  percentage/ 
number  of  citizens  engaged  in  community  decisions  and  the  adopt¬ 
ion  and  utilization  of  specific  democratic  procedures  in 
community  decision-making)  operationalized  as  program  objectives. 
Content  objectives  were  not  identified  early  in  the  change 
process  in  order  for  them  to  remain  flexible  enough  and  evolve 
as  a  natural  response  to  meet  the  expressed  needs  of  the  community. 
In  this  way  the  design  could  assess  both  intended  and  unantici¬ 
pated  effects  that  could  evolve  during  the  implementation  of 
the  nutrition  program  by  the  Peace  Corps.  Although  the  out¬ 
come  of  the  nutrition  program  appeared  on  the  surface  to  reveal 
negative  results,  an  assessment  of  what  happened  during  its 
process  revealed  the  following  important,  unanticipated  positive 
results  laid  out  by  Daley  and  Winter  (1978:  72-72) .  .  . 

1.  While  engaged  primarily  in  the  nutrition  program, 
volunteers  became  involved  in  other  areas  (e.g.  agriculture, 
education,  community  utilities)  all  of  which  proved  to  benefit 
from  the  presence  of  Peace  Corp  members. 
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2.  Peace  Corps  volunteer  presence  contributed  to 
limited  changes  in  resource  allocation  within  the  community. 

3.  Increased  participation  in  community  decision¬ 
making  and  co-operation  was  due  to  the  efforts  of  volunteers. 

Thus,  these  "spin-off"  or  unanticipated  products  of  volunteer 
efforts  which  were  revealed  through  a  process  evaluation 
became  the  most  important  factors  on  community  impact  in 
the  eyes  of  volunteers  and  community  members,  as  an  evaluation 
of  process  illustrated  that  the  Peace  Corps  nutrition  program 
had  a  successful  impact  on  individuals  and  institutions  in  the 
communities  surveyed.  In  short,  as  often  there  is  no  correlation 
between  a  program's  outcome  and  process,  unanticipated  con¬ 
sequences  must  be  considered  in  assessing  the  impact  of  a 
program.  Community  development  evaluators  should  continue  to 
emphasize  the  evaluation  of  process  through  descriptive  research 
designs . 

Inductive,  descriptive  designs  are  used  in  accordance 
with  the  systems  model  approach.  Lorion  (1977)  has  already 
defined  the  theoretical  premises  of  the  community  development 
process  as  being  founded  in  the  same  basic  principles,  values, 
and  beliefs  of  the  systems  model.  According  to  Lorion  (1977:22) 
this  approach  *  •  • 

1.  Assumes  that  the  community  as  an  environment,  sense 
of  belongingness,  or  however  defined, shapes  an  individual's 
behaviour  in  both  positive  and  negative  ways  providing  resources 
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for  adequate  coping  as  well  as  imposing  forms  of  stress  and 
restricting  the  range  of  alternative  solutions. 

2.  Assumes  that  within  a  community  there  exists  an 
identif icable  set  of  systems  (e.g.  education,  social  services, 
legal,  judicial,  political,  recreational)  affecting  positively 
and  negatively  psychological  states  of  its  members. 

3.  Assumes  that  communities  and  their  relevant  systems 
proceed  through  identifiable  stages  and  transitory  forms, 
which  can  be  considered  systematically  in  the  design  and 
implementation  of  interventions. 

4.  Assumes  that  involvement  in  communities  and  the 
systems  requires  a  synergistic  exchange  between  conceptualization 
and  program  implementation  such  that  each  develops  through 

the  efforts  of  the  other. 

The  theoretical  premises  provided  by  Lorion's  systems  model 
equips  the  community  development  practitioner  with  the  research 
base  for  improving  ways  of  planning,  implementing  and  evaluating 
systematic  changes  in  communities  and  their  institutions. 

A  notable  contribution  using  a  descriptive  and  systems 
approach  comes  from  Campfens  (1975)  who  advances  a  social  policy/ 
community  development  framework  which  provides  a  systematic 
approach  to  social  policy  analysis  in  order  to  clarify  the 
conceptual  and  operational  issues  inherent  in  the  relationship 
between  these  two  basic  concepts. 


Campfens  identifies  "the  unadulterated  economic  cost- 
benefit"  view  of  industrial  man  as  underlying  the  majority  of 
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policy/planning  approaches.  He  stresses  a  communal  view  of 
man  which  sees  man  in  terms  of  his  psycho-social  needs  and 
quality  of  life  issues.  This  latter  view  shows  more  respect 
for  man  and  human  values  and  in  this  way  allies  with  the  central 
concerns  of  community  development.  Campfens  utilizes  the 
theoretical  premises  of  the  communal  view  of  man  to  formulate 
an  alternative  to  the  policy/planning  models  in  order  to 
emphasize  the  community  as  the  organizational  focus  for  meeting 
human  needs  and  the  focal  point  at  which  relevant  community 
groups  can  become  involved  in  the  decision-making  of  the  policy 
formulation  and  planning  processes  at  all  societal  levels.  He 
lays  out  prime  intervention  strategies  and  targets  for  evaluation 
research  studies  in  community  development  and  urges  that 
community  development  be  viewed  in  more  realistic  terms  "as 
an  essential  part  of  society's  total  response  to  human  need 
and  social  problems."  Next,  Campfens  addresses  in  turn  each 
of  the  weaknesses  of  evaluation  research  in  community  develop¬ 
ment  which  are  its  absence  from  adequately  developed  social 
development  policies,  government  commitment  and  adequate  changes 
in  the  decision-making  system.  Campfens  concludes  that  from 
his  framework  of  social  policy/community  development  can  evolve 
a  systematic  evaluation  process  which*  first,  considers  the 
individual  in  society  rather  than  industrial  man  and  thus 
assumes  more  fully  the  community  as  the  organizing  principle 
for  social  policy;  second,  that  community  development  as  a 
problem-solving  activity  pursued  at  the  local  level  can  tie 
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into  the  decision-making  structure  and  processes  within  all 
society  generally;  and  third,  recognizes  that  the  activities 
of  social  policy  should  include  participation  by  both  the 
specialist  and  the  local  community  groups;  and  finally,  demon¬ 
strates  that  the  specialist  in  social  policy  practice  can  be 
an  effective  link  between  legislative  and  community  groups 
both  of  whom  must  have  an  effective  amount  of  input  into  the 
decision-making  system  which  allocates  resources  and  defines 
policies  which  affect  their  welfare. 

Thus,  we  have  seen  from  the  preceeding  analysis  that 
there  is  no  longer  an  absence  of  descriptive  designs  or  dynamic 
models  of  the  community  development  process.  Rather,  a  commun¬ 
ity  development  practitioner  can  utilize  those  outlined  in  this 
section  such  as  the  theory  based  approach,  systems  approach, 
or  social  policy/community  development  models  in  order  to 
guide  him  in  his  research  design  to  more  wholistically  examine 
task  and  process  objectives  and  the  theoretical  basis  of 
community  development  programs. 

We  have  examined  in  the  previous  section  the  use  of 
descriptive  designs  and  dynamic  models  such  as  theory  based  or 
systems  approaches  and  established  their  utility  in  conducting 
process  evaluations  and  handling  the  theoretical  premises  of 
the  community  development  process.  However,  in  some  instances 
the  use  of  experimental  or  quasi  experimental  designs  can  enable 
the  community  theorist  to  test  hypotheses  which  would  otherwise 
not  be  testable  since  rarely  can  community  variables  be 


113 


manipulated  for  purposes  of  research.  Voth  (1975:  154)  stated 
that  the  evaluation  researcher  of  community  development  programs 
has  in  the  past  been  unable  to  control  assignment  to  experi¬ 
mental  and  control  groups.  However,  a  review  of  the  most 
recent  literature  reveals  that  significant  attempts  at  utilizing 
such  empirically  orientated  studies  have  been  made  in  evalu¬ 
ating  performance  studies  of  community  workers. 

Nicasto  (1976:  73)  noted  that  in  the  past  efforts  to 
evaluate  community  development  training  programs  based  on  per¬ 
formance  criteria  have  lacked  the  rigorous  methods  of  experi¬ 
mental  design  (e.g.  Richard  Franklin  (1971),  Western  Regional 
Resource  Development  Committee  (1972)).  Nicasto  goes  beyond 
these  earlier  studies  which  merely  established  performance 
criteria  and  measures  and  attempted  to  apply  empirical  research 
using  standardized  instruments  in  order  to  define  and  differen¬ 
tiate  the  field  of  community  development  from  other  profession¬ 
al  helping  occupations.  Using  the  Strong-Campbell  Interest 
Inventory,  Nicasto  compared  a  group  of  84  community  developers 
to  67  existing  male  occupational  profiles.  Results  showed  that 
the  community  developers  were  distinct  from  others  in  sixty- 
seven  instances,  thereby  explicitly  showing  that  community 
development  practitioners  held  their  own  distinctive  character¬ 
istics  that  could  be  subjected  to  more  sophisticated,  system¬ 
atic  performance  evaluation  research  studies. 

Bannon,  (1977)  used  experimental  design  and  control 
groups  to  systematically  evaluate  community  development  leaders 
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trained  under  the  Roving  Recreation  Leader  Training  Guide  as 
compared  to  those  trained  in  traditional  programs  or  those 
with  no  training  at  all.  Bannon  developed  indicators  to 
measure  four  sets  of  criteria  (Figure  12).  Results  and  conclu¬ 
sions  from  his  rigorous  design  indicated  that  community  develop¬ 
ers  were  better  workers  when  they  underwent  the  Training  Guide 
Program.  Moreover,  Bannon  recommends  that  more  adequate 
measures  to  evaluate  the  job  performance  of  community  develop¬ 
ment  workers  along  with  more  precise,  accurate  data  sources 
such  as  true/false  and  multiple  choice  tests.  However,  in 
light  of  the  preceeding  examination  of  evaluating  community 
development  performance  programs  in  community  development 
through  empirically  orientated  designs,  we  can  conclude  that 
in  some  instances  the  use  of  the  experimental  design  can  be 
as  useful  a  research  technique  for  community  developers  as 
the  descriptive,  dynamic  designs.  Moreover,  community 
development  practitioners  should  not  as  Lafleur  contended 
(1977:  98)  restrict  themselves  to  the  last  three  levels  of 
Ross's  hierarchy  identified  as .  .  . 

1.  Classical  Fisherian  experiments,  preferably  with 
factoral  designs; 

2.  Quasi-experiments  with  impure  control  groups  (e.g. 
training  program  candidates  compared  with  unemployed  friends) ; 

3.  Correlational  designs  in  which  statistical  controls 


are  employed; 
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Figure  12 


Research  design  for  the  evaluation  of  the  Outreach 
Training  Program  (From  Bannon ,  1977) 
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4.  Audits  of  programs  and  projects,  employing 
qualitative  judgements  by  outside  observers; 

5.  Narrative  reports  of  program  administrators. 

In  short,  the  author  of  this  thesis  is  not  suggesting  in  any 
way  that  traditional  research  methods  (e.g.  experimental  design) 
are  always  appropriate  for  community  development  evaluations. 

On  the  contrary,  the  author  of  this  thesis  contends  that  in  a 
majority  of  cases  community  development  programs  are  more 
appropriately  evaluated  through  descriptive,  dynamic  designs; 
but  in  some  instances  the  community  development  practitioner 
should  and  can  use  the  experimental  model  in  conjunction  with 
precise  measuring  devices  and  thereby  implementing  a  reasonably 
powerful  research  design. 

In  summary,  the  process  (formative)  evaluation  level 
is  as  demanding  and  time  consuming  as  it  is  in  other  related 
social  development  fields.  Community  development  practitioners 
are  responsible  for  defining  both  quantitative  and  especially 
qualitative  measures  to  evaluate  the  achievement  of  task  and 
process  objectives.  In  a  majority  of  cases,  the  use  of  descrip¬ 
tive  research  designs  in  accordance  with  the  systems-model 
approach  appears  to  be  most  appropriate  for  investigating  the 
process  whereby  qualitative  change  occurs.  Moreover,  the 
flexibility  of  descriptive  and  dynamic  designs  reveal  unantici¬ 
pated  partial  results  which  are  important  in  determining  the 
success  or  failure  of  a  community  development  program.  Finally,  the 
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aversion  towards  utilizing  the  experimental  design  appears 
to  be  lessening  as  community  development  practitioners  are 
recognizing  that  it  can  serve  as  a  useful  design  for  studying 
the  changes  effected  by  community  development  programs. 

PRODUCT  EVALUATION  LEVEL 

The  product  evaluation  level  is  characterized  by  the 
activities  of  documentation  and  outcome  (summative) .  Moreover, 
the  oreceeding  section  has  emphasized  the  importance  of  process 
(formative)  evaluation  studies  in  community  development  as 
most  programs  are  ongoing  and  not  yet  complete.  However, 

-£-^g£-g  exists  in  the  literature  two  significant  outcome  studies 
(Voth  and  Martin,  1974  and  Voth,  1975)  which  contribute  towards 
a  more  systematic  accumulation  and  utilization  of  secondary 
data  that  has  been  greatly  lacking  and  hence  inhibiting  the 
development  of  comparative  studies  for  community  development. 

It  is  the  second  study  which  will  be  examined  as  an  illustration 
of  how  useful  small  samples  taken  from  a  population  of  groups 
or  communities,  can  be  used  in  conjunction  with  precise  measur¬ 
ing  devices  and  thereby  create  a  reasonably  powerful 
research  design  for  an  outcome  study  in  community  development. 

Donald  Voth  (1975:  635-647)  evaluated  the  impact  of 
29  community  development  programs  in  Southern  Illinois.  Voth 
used  a  probability  sample  of  29  program  and  32  non-program 
communities  to  test  the  hypothesis  that  community  development 
programs  have  an  effect  upon  process  and  content  variables  at 
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the  community  level.  The  process  variables  used,  were  voting 
participation  and  the  number  of  candidates  running  for  local 
elections.  Content  variables  included  four  cumulative, 
unidimensional  scales  of  the  institutional  structure  within 
the  communities,  namely  retail  services,  rural  services,  non 
economic  services  and  health  services.  Multiple  regression 
techniques  were  used  to  test  the  hypothesis  and  Scalogram 
Analysis  was  used  to  create  the  scales.  Findings  indicated 
that  community  development  programs  are  correlated  with 
improvements  in  elite  participation  and  in  the  scale  of  rural 
non-economic  and  health  services.  The  evaluation  illustrated 
that  the  29  community  development  programs  probably  did  have 
an  effect  on  the  services  available  in  the  target  communities 
but  the  effects  on  participation  were  only  partial.  Clearly 
Voth ' s  outcome  study  illustrates  the  ability  of  the  community 
development  practitioner  to  overcome  crude  measures,  utilize 
small  samples  effectively  in  a  sound  research  design,  and  by 
making  systematic  accumulation  and  use  of  secondary  data, 
contribute  to  an  increase  in  comparative  studies  in  community 

development . 

CONTEXT  EVALUATION  LEVEL 

As  Voth  (1975:  157)  stated  program  goals  and  objectives 
serve  different  functions  for  different  people  involved  in  making 
evaluative  decisions  regarding  a  community  development  program 
and  often  discrepancies  over  them  have  led  to  an  adversary 
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relationship  between  administrator  and  researcher.  For 
example,  evaluators  are  often  unable  or  reluctant  to  provide 
administrators  with  feedback  or  that  which  is  provided  is  in 
overly  scientific  or  technical  jargon.  In  order  to  overcome 
these  obstacles  a  research  design  should  try  to  integrate 
collaborative  efforts  between  the  administrator  and  researcher 
in  deciding  on  program  objectives,  operational  measures  accept¬ 
able  to  both  parties  and  to  try  and  anticipate  consequences 
from  the  dissemination  of  evaluation  reports  in  order  that  they 
do  not  become  "used  for  some  partisan  purposes  by  politicians". 
In  other  words,  Voth  is  advocating  that  a  research  framework 
should  consider  the  issues  relating  to  the  organizational  and 
socio-political  contexts  affecting  the  nature  of  the  community 
development  program  and  its  subsequent  evaluation. 

A  significant  contribution  towards  overcoming  such 
political  problems  and  towards  carrying  out  evaluation  research 
that  is  more  closely  allied  to  the  community  development 
process  "applied  and  used  by  public  agency  professionals"  is 
founded  in  Burton's  Systems-Process  Approach  (1978). 

In  order  to  provide  a  guide  for  conceptualizing  evalu¬ 
ation  efforts  within  a  community  development  context,  the 
systems-process  approach  identifies  five  major  stages  within 
the  evaluation  process  and  discusses  the  issues  with  which 
the  evaluator  must  deal;  developing  the  evaluation  contract; 
specifying  the  program  system  and  negotiating  the  evaluation 
scenario;  evidence  collection;  making  evaluative  judgements; 
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and  utilization  and  dissemination  of  evaluation  results. 

Within  each  of  these  stages,  Burton  stresses  the  need  for 
interaction  and  negotiation  between  all  program  stakeholders. 
Significantly,  he  extends  the  term  "stakeholders"  to  include 
not  just  those  intimately  involved  in  the  evaluation  process 
(e.g.  administrator  and  researcher)  but  also  to  the  potential 
consumers  of  the  program  who  are  the  community  members.  In 
this  way,  the  systems  process  captures  the  basic  theoretical 
premises  of  the  community  development  process  whereby  citizens 
play  an  active  role  in  defining  and  meeting  their  own  needs. 
Clearly,  Burton's  approach  is  a  significant  starting  point 
for  developing  a  framework  which  more  realistically  assesses 
success  of  a  community  development  program  in  terms  of  all 
stakeholders  involved  in  the  various  subsystems  within  which 
a  community  development  program  operates. 

Voth  (1975:  159)  noted  that  a  political  problem  in 
community  development  research  was  the  poor  utilization  of 
evaluation  results  which  are  often  ignored  by  administrators. 
Or,  as  already  noted,  the  evaluators  are  frequently  unable  or 
reluctant  to  provide  administrators  with  feedback  or  that  which 
they  do  provide  is  in  overly  technical  and  scientific  jargon. 

In  order  to  provide  for  more  effective  dissemination  of  evalu¬ 
ation  results,  Burton  (1978)  contends  that  the  way  in  which 
results  are  prepared  and  delivered  depends  on  who  will  utilize 
or  receive  them  and  should  thus,  be  easily  understood  by  all 
program  stakeholders.  For  example,  evaluation  reports  to 
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program  participants  may  differ  from  those  delivered  to  program 
administrators  and  all  program  stakeholders  must  receive  the 
results  from  evaluators.  Also,  in  this  context,  Willis  Goudy 
and  Frederick  Wepprecht  (1977)  offer  an  interesting  way  of 
disseminating  reseach  data  to  residents  in  terms  of  "informa¬ 
tive  packages"  which  interested  community  citizens  can  utilize. 
In  this  way  a  joint  effort  is  made  between  researchers,  admini¬ 
strator,  and  community  residents  thereby  helping  to  eliminate 
conflict  and  reduce  political  problems. 

In  short,  it  appears  that  community  development 
practitioners  are  giving  increased  attention  to  the  issues 
relating  to  the  contextual  elements  revolving  around  and  within 
their  programs  in  order  to  make  for  a  more  complete  research 
framework . 

SUMMARY 

The  critical  appraisal  undertaken  in  this  chapter 
suggests  that  the  literature  from  the  past  two  decades  related 
to  community  development  programs  reflects  a  growing  interest 
in  their  evaluation.  And  moreover,  a  number  of  significant 
attempts  appear  to  have  been  made  towards  developing  and 
improving  evaluation  methodologies  through  systematic,  applied 
social  action  research.  In  this  way,  evaluation  in  community 
development  is  moving  towards  becoming  a  "science"  and  an 
"art"  and  such  a  trend  has  assisted  in  overcoming  the  difficult¬ 
ies  often  associated  with  conducting  evaluations  in  community 
development  for  at  least  the  reasons  set  down  by  Voth  (1975: 
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1.  Ambiguity  of  goals  in  community  development. 

2.  Absence  of  a  model  of  the  community  development 
process . 

3.  Inability  of  the  researcher  to  control  assignment 
to  treatments. 

4.  Weak  effects,  crude  measurement,  and  small  samples. 

5.  Political  problems — the  relationship  between 
evaluation  and  program  administration. 

It  appears  that  through  the  borrowing  and  modifying  of  those 
approaches  from  social  action  research  which  are  most  appli¬ 
cable  to  the  community  development  field,  the  following 
advancements  have  been  made.  First,  goals  have  become  less 
ambiguous  and  stated  in  operational  terms  through  clear  recognition 
of  two  sets  of  community  development  objectives,  namely  task  and 
concrete  goals.  In  consequence,  this  has  provided  for  more 
precise  objective  program  guides  and  sound  measures  for  evalu¬ 
ating  the  changes  which  community  developers  attempt  to  effect 
through  community  programs  in  both  quantitative  and  especially 
qualitative  terms.  Moreover,  the  conceptualization  of  program 
objectives  has  been  performed  in  accordance  with  community 
needs  assessments  which  provide  an  important  informational 
base  or  input  from  both  the  people  of  a  community  and  the 
practitioner.  Second,  there  appears  to  be  an  emphasis  on  assess¬ 
ing  ongoing  changes  created  by  community  development  programs 
as  they  reveal  unanticipated  partial  results  whose  significance 
should  be  considered  in  order  to  more  properly  evaluate  the 
success  of  a  program  whose  outcome  is  not  always  correlated 
with  its  process.  Third,  there  is  an  emphasis  on  the  use  of 
descriptive  designs  and  development  of  dynamic  causal  models 
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such  as  the  theory  based,  systems  and  community  development/ 
social  policy  approaches  which  have  served  as  important  guide¬ 
lines  for  the  researcher  in  assessing  more  wholistically  the 
effects  of  the  community  development  process.  Fourth,  no 
longer  is  there  an  aversion  towards  utilizing  the  experimental 
design  because  of  the  inability  of  the  researcher  to  assign 
control  groups,  which  allows  for  more  systematic  and  objective 
evaluations.  Fifth,  the  development  of  precise  measures  and 
effective  use  of  small  group  samples  have  been  integrated  into 
reasonably  powerful  research  designs  for  outcome  studies  in 
community  development.  Finally,  the  political  atmosphere  within 
which  evaluative  decisions  regarding  community  development 
programs  are  made  has  become  more  compatible  towards  develop¬ 
ing  collaborative  efforts  between  all  program  stakeholders, 
namely  the  researcher,  administrator  and  consumers  of  the 
community  development  program. 

However,  despite  this  significant  progress  there  remains 
a  lack  of  systematic  evaluation  frameworks  and  procedures 
containing  common  classifications,  criteria  and  indices  related 
to  community  development.  In  order  to  fill  this  serious  void, 
the  final  aim  of  this  thesis  is  to  design  a  simplified  evalu¬ 
ation  framework  for  practitioners  in  this  field.  It  will  combine 
first,  the  conceptual  groundwork  provided  from  "Evaluation  as 
an  Art"  which  depicted  leading  statements,  principles,  criteria 
and  approaches  related  to  community  development  evaluations; 
and  second,  those  practical  procedures  of  social  action  research 
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reviewed  in  chapter  2  whose  relationship  and  applicability  to 
community  development  was  appraised  in  "Evaluation  as  a  Science" . 
The  synthesis  and  overview  of  this  material  will  provide  the 
guidelines  towards  designing  an  evaluation  framework  which 
will  hopefully  assist  in  improving  the  state  of  evaluation 
practices  in  community  development  programs. 


CHAPTER  4 


A  SIMPLIFIED  FRAMEWORK  FOR  THE  EVALUATION 
OF  COMMUNITY  DEVELOPMENT  PROGRAMS: 

A  SYSTEMS-PROCESS-CONTEXT  APPROACH 

Synthesis  and  Overview 

At  this  point,  it  is  useful  to  provide  a  synthesis 
and  overview  of  the  material  covered  in  the  preceeding 
chapters  in  order  to  clarify  the  basic  premises  constituting 
the  framework  developed  within  this  chapter. 

The  distinguishing  characteristics  of  community  develop 
ment  programs  indicate  that  a  framework  for  evaluation  in  this 
field  should  be  one  that  will: 

1.  Integrate  the  important  components  of  all  four 
evaluation  levels — input,  process,  product,  and  context — and 
their  corollary  evaluation  activities  (pre-f ormative ,  formative 
summative)  into  a  systems  model  approach.  In  this  way  will 

be  provided  an  overall  descriptive  study  design  for  operation¬ 
alizing  the  theoretical  units  of  the  program  to  provide  for  a 
more  systematic  evaluation. 

2.  Highlight  the  significance  of  both  quantitative 
and  qualitative  changes/dimensions  in  terms  of  identified 
task  and  process  objectives,  whose  conceptualization  is 
achieved  in  accordance  with  the  theoretical  premises  of  the 
overall  community  development  program  and  the  informational 
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input  provided  from  a  community  needs  assessment. 

3.  Reduce  the  dimensions  of  task  and  process  objectives 
into  identifiable  elements  reflecting  the  nature  of  such 
objectives  in  both  quantitative  and  especially  qualitative 
terms . 

4.  Focus  on  the  evaluation  of  process  objectives  and 
define  their  elements  in  relation  to  changes  to  be  effected 
to  the  target  community  and/or  individual  and  on  changes  to 
be  effected  on  contextual  factors  (e.g.  administrative  and 
political) .  In  addition,  it  should  conceptualize  the  elements  of 
process  objectives  in  a  hierarchical  scheme  based  on  immediate, 
intermediate  and  ultimate  goals  in  order  to  make  objectives 
more  operational  and  subject  to  more  precise  measurements. 

5.  Develop  both  quantitative  and  qualitative  measures 
to  assess  the  elements  characteristic  of  both  the  task  and 
particularly  the  process  objectives. 

6.  Emphasize  in  the  framework  the  process  evaluation 
level  and  its  inherent  formative  evaluation  activities  focussed 
upon  the  implementation  and  progress  of  a  program  towards 
reaching  its  task  and  process  objectives  through  the  specifica¬ 
tion  of  both  quantitative  and  qualitative  measures. 

In  short,  the  framework  tries  to  integrate  what  appears 
to  be  the  most  significant  aspects  of  social  action  research 
related  to  community  development  evaluations  into  a  systems- 
process-context  approach. 
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Based  on  these  particular  characteristics  the  material 
in  this  chapter  will  follow  an  organizational  basis  which 
requires  an  undertaking  of  the  following  tasks:  first,  the 
major  evaluation  components  will  be  identified  as  the  four 
major  evaluation  levels  (input,  process,  product  and  context) 
and  their  related  evaluation  activities  within  the  pref ormative , 
formative  and  summative  evaluation  stages;  second,  the  dimensions 
of  the  primary  objectives  pursued  by  the  theoretical  premises 
of  a  community  development  program  will  be  identified  as  task 
and  process  objectives;  third,  a  description  of  the  character¬ 
istics  of  the  evaluative  dimensions  will  be  given  in  terms  of 
defining  elements  related  specifically  to  each  of  the  task  and 
especially  the  process  objectives;  and  fourth,  quantitative 
and  qualitative  measures  will  be  outlined  in  order  to  provide 
an  objective  and  precise  measuring  base  for  the  elements  within 
the  dimensions  of  task  and  process  objectives. 

A  typology  related  to  the  preceeding  four  tasks  for 
undertaking  an  evaluation  of  community  developments  program, 
has  been  illustrated  in  Figure  13.  The  terms,  components, 
dimensions,  elements  and  measures  depicted  in  the  typology 
have  been  borrowed  from  a  study  by  Homenuck,  Durlak  and 
Morgenstern  (1977  :  103-119)  though  the  typology  itself  is  not  theirs. 

The  framework  developed  in  this  thesis  is  meant  to  be 
a  useful  starting  point  for  understanding  what  occurs  in  a 
community  development  program  and  outlines  the  basic  steps  to 
assist  in  evaluating  its  effects  upon  program  recipients.  In 
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Figure  13  Typology  of  evaluation  tasks  related  to  coinminity  development  programs 
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this  way  it  is  a  learning  framework  whereby  one  can  discover 
how  to  avoid  mistakes  and  thereby  improve  the  process  as 
utilized  in  programs  with  community  development  orientation. 

The  purpose  is  to  improve  the  technical  quality  of  conducting 
community  development  evaluations  by  providing  a  systematic 
and  structured  approach  that  will  aid  in  determining  the 
legitimacy  of  community  development  programs,  to  generally 
upgrade  the  quality  of  community  development  evaluation  reports, 
and  most  importantly  to  demonstrate  accountability,  since 
clearly  as  monies  become  scarce  community  development  practi¬ 
tioners  will  become  increasingly  pressed  from  funding  sources 
to  demonstrate  accountability.  One  of  the  best  ways  to  achieve 
this  is  through  systematic  evaluation  approaches  based  on  the 
principles  of  applied  social  action  research. 

The  framework  developed  in  this  chapter  is  a  simplified 
and  general  one.  That  is  to  say,  this  thesis  will  not  concern 
itself  with  the  application  of  this  framework  to  a  field 
project  or  actual  case  study.  The  purpose  is  limited  to  the 
development  of  a  practical  framework  which  can  be  tested  and 
applied  by  others  in  the  community  development  field  (e.g. 
government  or  quasi-government  agencies)  which  will  aid  in 
improving  the  state  of  evaluation  practice  in  the  field. 
Moreover,  the  framework  has  maintained  flexibility  so  that  it 
can  be  adjusted  to  meet  the  specific  requirements  of  a 
community  group  or  practitioner  responsible  for  the  evaluation 
of  community  development  programs  in  urban  settings. 


For 
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example,  the  framework  has  been  developed  in  conjunction  with 
the  efforts  of  researchers  at  the  Edmonton  Social  Planning 
Council  so  that  such  a  community  development  agency  can  utilize 
aspects  of  its  conceptual  basis  to  assist  in  undertaking  a 
practical  evaluation  of  the  programs  it  supports. 

THE  FRAMEWORK 

THE  SYSTEMS-PROCESS-CONTEXT  APPROACH 

Components  of  Evaluation 

The  major  components  in  evaluating  community  development 
programs  can  be  identified  in  relation  to  the  four  evaluation 
levels  and  corollary  evaluation  activities  undertaken  in  the 
evaluation  of  other  social  action  programs,  which  have  been 
noted  throughout  the  thesis. 

These  major  components  inherent  within  a  community 
development  evaluation  have  been  synchronized  and  synthesized 
with  a  systems  model  approach  illustrated  in  Figure  14.  As 
shown,  this  perspective  permits  constant  feedback  by  the 
administrator  on  perceiving  a  program's  success  in  terms  of 
both  quantitative  and  qualitative  changes  effected  upon  an 
individual  or  community  group.  Figure  14  also  illustrates 
and  makes  explicit  the  following.  First,  the  community  develop¬ 
ment  program  is  evaluated  in  terms  of  decisions  made  during  the 
preformative  evaluation  stages,  defined  here  as  program  planning 
and  community  needs  assessment.  Together,  these  two  activities 
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provide  the  informational  base  or  input  into  the  conceptuali¬ 
zation  and  measurement  of  program  objectives  based  on  the 
felt  needs  of  both  the  community  and  the  administrator. 

Second,  the  community  development  program  should  also  be 
evaluated  in  terms  of  judgemental  decisions  made  during  the 
formative  evaluation  stage  where  the  emphasis  is  on  examining 
the  progress  of  the  program  during  its  implementation  rather 
than  at  its  end.  In  this  way,  emphasis  is  placed  on  an  ongoing 
assessment  of  the  program's  process  towards  reaching  the  over¬ 
all  program  objectives,  specifying  the  qualitative  and  quanti¬ 
tative  measures,  and  choosing  the  appropriate  research  design. 
Third,  decisions  where  possible,  can  be  made  towards  judging 
the  product  or  outcome  of  a  community  development  program 
(summative  evaluations).  Finally,  a  consideration  of  context¬ 
ual  factors  influencing  a  community  development  program  must 
be  given  consideration  based  on  the  assumption  that  the 
hypothesis,  strategies  and  likely  outcomes  of  an  evaluation 
study  are  affected  by  such  factors.  Hence,  the  consideration 
of  context  helps  to  provide  more  realistic  decisions  made  at 
the  input,  process  and  product  evaluation  levels  (e.g.  defining 
elements  of  dimensions  in  the  formative  evaluation  stage) . 

In  short,  this  dynamic  causal  diagram  operationalizes 
the  major  components  of  the  community  development  evaluation 
process,  generating  hypothetically  for  the  researcher  a  number 
of  theoretical  units  which  describe  the  interactions  between 
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these  units  and  thereby  provides  an  overall  descriptive  study 
design  for  the  undertaking  of  a  systematic  evaluation  approach 
to  community  development  programs.  It  is  the  components  of 
the  process  evaluation  level  and  its  corollary  formative 
evaluation  activities  of  implementation  and  progress  towards 
assessing  quantitative  and  qualitative  changes/dimensions 
which  will  be  the  focus  of  this  framework. 

Dimensions  of  Evaluation  Objectives 

and  Related  Elements 

A  consensus  from  the  appraisal  of  evaluation  literature 
in  community  development  has  established  that  its  program 
objectives  are  to  create  quantitative  and  more  importantly 
qualitative  change  in  the  individual  and/or  community.  More¬ 
over,  there  are  two  distinct  program  objectives,  whose  dimen¬ 
sions  can  be  defined  as  task  and  process.  Figure  15  illustrates 
that  their  conceptualization  has  been  defined  in  accordance 
with  the  theoretical  premises  of  the  community  development 
process.  That  is,  both  objectives  are  merely  two  different 
change  approaches  in  the  overall  community  development  program 
towards  reaching  the  same  ultimate  goal  of  improving  the 
quality  of  life  for  the  individual  and/or  community. 

Task  objectives  refer  to  changes  in  social/physical 
environments  such  as  facilities  and  services.  These  results 
are  concrete,  tangible  and  easy  to  measure  in  quantitative 
terms.  In  contrast  process  objectives,  referring  to  socio-psychol- 
ogical  development  in  the  individual  or  collective  changes  in 
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the  behaviour  of  participants  engaged  in  a  community  develop¬ 
ment  program  is  more  difficult  to  measure  quantitatively  and 
they  require  the  development  of  qualitative  measures  to 
assess  their  effects.  Moreover,  the  appraisal  of  evaluation 
literature  related  to  community  development  has  indicated 
an  emphasis  upon  the  definition  and  measurement  of  process 
objectives  and  hence,  they  will  be  a  focal  point  in  this  framework. 

Within  each  dimension  exists  a  group  of  more  specific 
task/process  requirements  referred  to  as  elements  of  the 
dimensions.  The  next  step  then  becomes  to  identify  an  addit¬ 
ional  level  of  complexity;  to  delineate  each  dimension  of 
task  and  especially,  process  objectives  as  they  are  the  focal 
point  of  community  development  evaluations,  into  a  number  of 
related  elements  which  can  be  later  assigned  quantitative  and 
qualitative  measures. 

Table  l,p.  136  lists  the  elements  of  task  as  services 
and  facilities.  Specifically,  services  can  be  either  direct, 
such  as  those  provided  by  formalized  institutions  touching 
upon  areas  such  as  health,  social  welfare,  judicial,  recreation. 
Indirect  services  are  those  which  have  not  been  highly  formal¬ 
ized  by  institutions  or  directed  at  a  specific  clientele  and 
it  is  the  latter  group  which  are  the  main  focus  of  community 
development  as  they  reflect  self-help  schemes  such  as  cooperatives, 
day  care  centers,  and  a  host  of  services  supplied  by  volunteer 
agencies  designed  to  meet  the  felt  needs  of  program  participants. 
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Facilities  are  very  interrelated  with  services  as 
often  one  cannot  provide  a  service  unless  there  is  a  facility 
in  existence;  a  partial  facility  may  exist,  but  until  its 
utility  is  demonstrated  in  the  community  it  will  not  be 
expanded  (e.g.  a  single  service  centre  must  demonstrate  its 
utility  before  it  is  expanded  into  a  multi-service  unit) . 

Table  1,  p. 136  lists  the  elements  of  process  as  the 
following : 

1.  Changes  in  administrative  decision-making  processes. 

Description  of  Objective.  To  develop  within  the 

practitioner  an  "innovative"  orientation  to  administration 
whereby  he  works  with  the  people  by  helping  them  to  participate 
in  defining  their  own  needs  and  the  ways  to  meet  them.  In 
this  way  collaborative  efforts  are  created  between  the  "practi¬ 
tioner"  and  community  citizens  ensuring  that  the  program  will 
maintain  its  community  development  orientation. 

2.  Enhancement  of  the  individual's  personal  condition. 

Description  of  Objective.  To  improve  the  socio- 

psychological  conditions  in  the  individual,  family  or  community 
group  and  thereby  improve  the  overall  quality  of  life. 

3.  Improved  sense  of  community  identity. 

Description  of  Objective.  To  develop  an  increased 

awareness  of  felt  needs,  community  problems  and  issues  within 
both  the  individual  and  collective  community  body.  These 
developments  will  ultimately  lead  citizens  to  utilize  the 
available  resources  to  create  self-help  and  self-reliant  schemes 
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to  meet  the  service  needs  of  the  community. 

4.  Improved  local  decision-making  processes. 

Description  of  Objective.  To  increase  the  decision¬ 
making  powers  of  the  people  at  the  local  level  which  will 
ultimately  lead  to  a  de-centralization  of  decision-making  in¬ 
fluences  by  outside  power  groups.  In  this  way,  the  community 
will  organize  its  collaborative  efforts  to  make  its  own 
"action"  strategies  to  meet  outside  influences  imposed  on  it. 

5.  The  ability  to  maintain  a  viable  community  in 
relation  to  the  larger  environment. 

Description  of  Objective.  To  develop  a  recognition  of 
the  setting  in  which  policies  are  made  and  to  educate  the 
community  members  in  these  important  areas.  Ultimately,  this 
will  enhance  the  ability  of  the  community  to  communicate  with 
larger  environments  (e.g.  local  government)  and  other 
communities . 

6.  Increased  government  and/or  political  commitment. 

Description  of  Objective.  To  acquire  financial  support 

from  larger  funding  bodies  (e.g.  government  or  non-government 
agencies).  Also,  to  acquire  political  support  from  groups 
within  the  community  (e.g.  local  power  groups  and  elites)  and 
outside  the  local  level  (e.g.  city  politicians,  bureaucrats, 
city  planners). 

Figure  16  illustrates  the  conceptualization  of  elements 
of  process  objectives  arranged  in  the  form  of  a  hierarchical 
scheme  ranking  several  cognition  levels  of  prime  and  supporting 
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FIGURE  16  Hierarchy  of  the  immediate,  intermediate  and  ultimate  elements 
of  process  objectives  in  community  development  programs 
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activities  based  on  immediate ,  intermediate  and  ultimate  aims. 

In  reviewing  the  spectrum  of  elements  possibly  related 
to  process  objectives,  it  was  found  that  to  date,  the  efforts 
of  most  evaluation  studies  in  community  development  have 
been  confined  primarily  to  defining  elements  of  process 
objectives  related  to  assessing  the  effects  of  changes  on  the 
target  community  or  individual  partaking  in  a  program. 

Although  these  elements  are  an  important  part  of  an  evaluation 
framework  in  community  development  they  should  not  constitute 
the  entire  basis  for  evaluation.  Rather,  elements  must  relate 
to  assessing  the  effects  of  change  on  contextual  factors  that 
can  affect  the  manner  in  which  a  community  development  program 
is  delivered. 

Contextual  factors  as  illustrated  in  Figure  17,  refer 
to  the  "social  ecology"  or  those  economic,  social,  cultural, 
pedagogical/organizational  and  political  factors  that  are 
involved  in  the  agency  interfacing  with  the  recipients  of 
the  program  in  the  operation  and  maintenance  of  programs  that 
are  appropriate  to  community  circumstances.  Thus,  in  addition 
to  identifying  elements  of  process  objectives  directly  related 
to  creating  changes  in  the  individual  and/or  community  group 
(e.g.  enhancement  of  individual  personal  condition,  improved 
sense  of  community  identity,  improved  local  decision-making 
processes  and  maintaining  a  viable  community  in  relation  to  the 
larger  environment) ,  this  framework  has  also  identified  those 
contexual  elements  as  first,  changes  in  administrative  decision- 
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FIGURE  17  Contextual  elements  related  to  process 

objectives  in  community  development  programs 
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making  processes;  and  second,  increased  government  and/or 
political  commitment.  These  contextual  elements  must  be 
measured  in  order  to  more  objectively  and  realistically  assess 
the  success  or  failure  of  a  program  in  community  development. 

Evaluation  Measures 

The  specification  of  measures  for  the  evaluation 
of  community  development  programs  follows  directly  from  the 
identification  of  the  elements  related  to  task  and  process 
objectives . 

Two  types  of  measures  have  been  referred  to  throughout 
this  thesis:  quantitative  and  qualitative,  the  latter  of  which 
are  subjective  social  indicators  which  measure  how  an  individual 
or  group  perceives  the  situation  within  the  program  in  which 
they  are  participating.  Such  subjective  indicators  assist  in 
the  relative  ranking,  clarification  and  significance  of  the 
quantitative  measures  and  thus,  they  are  not  an  end  in  them¬ 
selves  (Henderson,  1974).  Qualitative  measures  are  more 
intuitive  and  based  on  the  experience  and  expertise  of  the 
evaluator.  They  are  less  "scientific"  and  could  be  transformed 
quantitatively  but  this  is  expensive  and  time  consuming  and 
unnecessary  based  on  the  premise  that  a  necessary  and  important 
aspect  of  the  evaluation  process  is  to  make  informed  judgements 
particularly  with  regard  to  assessing  qualitative  changes. 

A  list  of  both  qualitative  and  quantitative  measures 
for  each  dimension  of  task  and  process  objectives  which  can 
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exist  within  a  community  development  program  is  given  in 
Table  2. 

In  Table  2,  pp.  144-150  are  designated  the  dimensions 
of  task  and  process  and  below  each  category  are  listed  the 
elements  related  to  the  particular  set  of  objectives.  In 
addition,  the  items  to  be  considered  in  measuring  the  elements 
are  described  and  listed  in  the  form  of  questions  in  column  2. 
Finally,  the  nature  of  the  measure  is  described  as  quantitative 
qualitative  or  in  cases  when  both  types  of  measures  appear  then 
the  term  "mixed"  has  been  utilized  in  column  3.  A  discussion 
of  a  few  examples  of  the  measures  related  to  each  of  the 
elements  of  task  (services  and  facilities)  and  process  dimen¬ 
sions  will  now  be  outlined. 

Recipients  of  a  service  (Table  2,  p.  144, b)  are  relat¬ 
ively  easy  to  describe  as  they  can  include  senior  citizens, 
youth  or  the  various  groups  of  disadvantaged  people  residing  in 
a  community.  Data  compiled  from  records  on  the  numbers  of  people 
making  use  of  a  service  are  easily  quantified  and  indicate  the 
degree  of  involvement  of  people  in  the  service.  How  accessible 
the  service  is  made  to  community  members  can  be  determined  by 
the  ways  and  degree  to  which  it  is  widely  publicized  (e.g. 
flyers,  advertisements  in  community  newsletters,  radio  announce 
ments)  and  as  this  measure  requires  an  amount  of  judgement 
on  the  part  of  the  evaluator  it  is  qualitative  in  nature. 

The  money  spent  on  the  building  of  a  facility  (Table 
2,  p.  145, e)  and  the  costs  to  maintain  its  upkeep  are  easily 
quanitified.  Costs  of  a  facility  relative  to  the  amount  of 
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use  made  of  it  by  a  service  organization  can  also  be  consid¬ 
ered  in  an  assessment. 

Changes  in  administrative  decision-making  process 
(Table  2,  p.  146)  implies  that  for  a  program  to  maintain  a  true 
community  development  orientation,  the  administrator  must 
maintain  an  "innovative"  approach  of  working  with  people 
rather  than  for  people.  He  must  remain  sensitive  to  people's 
needs  by  allowing  them  access  and  input  into  the  decision-making 
processes  for  determining  what  programs  a  community  needs  and 
wants.  The  priorizing  of  programs  prior  to  their  implementation 
can  be  determined  through  the  information  collected  by  citizen 
surveys,  community  forums  and  interviews.  The  data  collected 
from  these  sources  are  easily  quantified , (Table  2,  p.  146a). 

Who  makes  the  decision  at  the  local  level  (Table  2, 
p.l48a)  can  be  determined  by  measuring  the  degree  to  which  local 
leaders  or  dominant  groups  (e.g.  upper  classes)  exist  in  a 
community  and  the  amount  of  influence  they  have  in  directing 
the  decisions  made  by  the  rest  of  the  community  members. 

In  short,  Table  2  has  outlined  the  dimensions  of  task 
and  process  and  assigned  to  these  two  sets  of  objectives 
their  related  elements  which  describe  the  quantitative  and 
qualitative  changes  to  be  effected  in  the  individual  and/or 
community  group  and  in  contextual  factors  internal  or  external 
to  the  community  development  program.  Examples  or  quantitative 
and  qualitative  measures  considered  to  be  appropriate  in 
evaluating  a  community  development  program  were  also  specified. 
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The  list  of  measures  is  not  intended  to  be  all  inclusive,  but 
to  provide  the  evaluator  with  a  more  precise  measuring  base 
for  conducting  a  systematic  evaluation  of  the  effects  from 
community  development  programs. 

SUMMARY 

The  framework  outlined  in  this  chapter  has  been 
designed  for  the  evaluation  of  community  development  programs 
and  its  design  has  been  based  on  those  principles  of  social 
action  research  which  appear  to  be  most  appropriate  for 
handling  the  distinguishing  characteristics  of  community 
development  programs . 

The  framework  identified  the  major  evaluation  components 
as  the  four  evaluation  levels  -  input,  process,  product,  context 
and  corollary  evaluation  activities  in  the  three  evaluation 
stages  (pre-f ormative ,  formative  and  summative) .  The  pro¬ 
cedures  of  the  major  evaluation  components  were  synchronized 
into  a  systems-model  approach  which  emphasizes  constant  feed¬ 
back  particularly  in  the  process  evaluation  level  and  its 
related  formative  evaluation  activities.  This  level  has  been 
the  focal  point  of  the  framework  in  order  to  assess  a  program's 
success  in  terms  of  effecting  both  qualitative  and  quantitative 
changes  upon  program  recipients.  Dimensions  of  community 
development  objectives  were  identified  as  task  and  process. 
Elements  related  to  task  and  especially  process  objectives 
were  identified,  described  and  subsequently  assigned 
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qualitative  and/or  qualitative  measures  in  order  to  assess 
a  program's  success  in  terms  of  both  qualitative  and  quantita¬ 
tive  changes  effected  upon  an  individual  and/or  community, 
and  also  upon  contextual  factors.  In  short,  the  emphasis 
has  been  on  the  utilization  of  the  systems  approach,  the 
process  evaluation  level  and  a  consideration  of  the  influence 
of  context  on  perceiving  the  program  as  a  success  or  failure. 
Hence,  the  framework  in  this  chapter  has  been  entitled  a 
systems -process -con text  approach. 

The  purpose  of  the  framework  is  threefold.  It  is 
hoped:  first,  that  this  approach  will  be  a  useful  starting  point 
towards  developing  more  systematic  evaluation  methodologies 
which  are  greatly  lacking  in  community  development  and  in 
this  way,  contribute  towards  improving  the  practice  of  evalu¬ 
ation  in  this  field;  second,  that  this  framework  will  be 
applied  to  actual  case  studies  and  the  results  of  the  assess¬ 
ment  will  be  reported  to  others  in  the  field,  as  it  is  only 
through  practical  experience  that  this  framework  can  be  modified 
and  improved;  and  third,  that  it  will  encourage  practitioners 
in  other  social  development  fields  related  to  community 
development  to  design  systematic  evaluation  procedures  approp¬ 
riate  for  handling  the  specific  nature  of  their  programs 
because,  as  Carter  and  Wharf  (1973:  80)  point  out,  it  has 
remained  a  vexing  problem  that  an  evaluation  framework  approp¬ 
riate  for  one  type  of  program  is  inapplicable  to  another. 
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CHAPTER  5 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  thesis  has  been  directed  at  examining  the  field 
of  evaluation  as  it  exists  in  theory  and  practice  in  social 
development  fields,  and  more  specifically,  to  ascertain  its 
relationship  to  community  development  in  order  to  develop  a 
framework  for  evaluating  programs  in  this  field.  From  this 
examination  can  be  drawn  the  following  conclusions  and 
recommendations . 

Conclusions 

It  has  become  very  apparent  that  the  field  of  evalu¬ 
ation  is  a  tremendously  complex  and  fascinating  area  within 
which  there  exists  in  inexhaustible  amount  of  literature.  It 
is  no  accident  that  the  field  of  evaluation  has  generated 
much  interest  over  the  past  decade  as  social  programs  are 
expanding  in  numbers  and  subsequently  there  has  arisen  a 
demand  from  funding  sources  upon  support  agencies  to  demon¬ 
strate  accountability  of  their  programs. 

The  field  of  community  development  is  feeling  the 
same  demands  for  accountability  from  its  funding  sources  as 
its  related  social  development  fields  but  it  still  lags  behind 
these  areas  in  providing  systematic  evaluation  approaches  with 
which  to  assess  the  effects  of  its  programs  upon  their 
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recipients.  In  light  of  this  serious  liability,  it  appears 
that  community  development  practitioners  over  the  past  decade 
have  attempted  to  generate  more  interest  in  evaluation 
studies  within  their  field. 

The  result  has  been  to  apply  the  principles  of  social 
action  research  towards  developing  evaluation  methodologies 
directly  related  to  assessing  the  effects  of  community  develop¬ 
ment  programs.  In  this  way,  the  field  of  community  develop¬ 
ment  is  providing  more  systematic  approaches  to  evaluations 
which  can  now  be  perceived  of  as  a  "science"  and  not  as  an 
"art".  However,  despite  this  significant  "advancement" 
community  development  practitioners  still  remain  reluctant 
to  commit  themselves  towards  defining  common  classifications, 
criteria,  and  indices  which  could  be  integrated  into  evaluation 
frameworks  specifically  designed  for  practical  use  in  community 
development  and  it  is  this  shortcoming  which  prompted  this 
present  thesis  to  begin  development,  tentatively,  of  such  a 
framework . 

The  framework  developed  in  this  thesis  focussed  on 
borrowing  and  modifying  from  the  knowledge  base  of  social 
action  research  those  principles  which  could  be  integrated 
into  an  approach  that  could  most  appropriately  handle  and 
evaluate  the  characteristics  of  community  development  programs. 
The  framework  emphasized  that  such  an  approach  should  encompass 
the  issues  in  what  appears  to  be  the  four  major  evaluation 
levels  -  input,  process,  product,  context  -  and  the  activities 
in  the  pre-f ormative ,  formative  and  summative  evaluation  stages. 
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In  this  way,  a  systematic  and  objective  framework  was  specifi¬ 
cally  developed  to  provide  useful  guidelines  for  the  practi¬ 
tioner  responsible  for  conducting  a  systematic  evaluation  of 
community  development  programs.  However,  as  the  systems- 
process-context  approach  is  the  first  of  its  kind  in  the  field 
of  community  development,  certain  limitations  can  inevitably 
be  drawn  from  its  conceptual  basis.  First,  the  emphasis  in 
the  framework  has  been  on  the  formative  activities  in  the 
process  evaluation  level  which  suggests  that  other  frameworks 
could  be  similarly  developed  to  investigate  more  thoroughly 
the  pre-f ormative  or  summative  evaluation  stages.  Second,  the 
system-process-context  approach  has  emphasized  a  descriptive 
research  design  which  may  fail  to  meet  the  needs  of  those 
requiring  an  experimental  design  for  undertaking  a  more  rigorous, 
statistically  orientated  analysis  and  therefore,  its  theoret¬ 
ical  premises  could  be  modified  to  meet  the  needs  of  such 
"scientifically"  orientated  researchers.  Finally,  the  frame¬ 
work  developed  in  this  thesis  is  a  static  model  which  handles 
a  dynamic  process  with  all  the  limitations  this  implies.  That 
is,  a  characteristics  of  evaluation  research  is  that  it  takes 
place  in  an  action  setting.  In  other  words,  something  else 
besides  research  is  going  on.  For  example,  the  institutional 
and  power  structures  existing  within  and  around  the  program 
can  influence  and  complicate  the  evaluation  process  which 
creates  a  dilemma  for  the  evaluator:  how  to  remain  close 
enough  to  such  influences  and  issues  from  the  program 
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environment  in  order  to  obtain  information  and  insights 
necessary  for  undertaking  a  sensitive  evaluation  while,  at 
the  same  time,  keeping  the  research  operation  autonomous 
enough  to  protect  its  objectivity.  The  solution  appears  to 
be  that,  although  the  systems-process-context  approach  is 
potentially  applicable  in  any  evaluative  situation  (internal 
and  external  evaluator)  this  framework  is  probably  best  used 
by  an  independent  evaluator  who  has  the  confidence  of  all 
parties  holding  some  interest  in  the  decision-making  processes 
surrounding  an  evaluation.  The  role  of  such  an  evaluator 
would  be  to  constantly  interact  and  negotiate  with  all  interest 
groups  in  the  identification,  specification  and  measurement  of 
program  objectives  as  held  acceptable  by  all  parties.  In 
this  way,  conflict  and  discrepancies  could  be  alleviated  and 
as  a  result,  any  possible  arising  interferences  and  disruptions 
for  research  operations  could  be  minimized. 

Recommendations 

The  following  recommendations  are  suggestions  for 
improving  the  state  of  the  evaluation  practice  in  the  field 
of  community  development.  The  author  of  this  thesis  recommends 
that : 

1.  The  framework  developed  in  this  thesis  be  applied 
to  case  studies  in  order  to  test  and  determine  its  applicability 
for  practical  use  in  the  evaluation  of  community  development 


programs . 
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2.  The  Division  of  Community  Development  at  the 
University  of  Alberta  require  that  its  M. A.  students  take 

a  mandatory  course  in  evaluation  research.  In  this  way,  the 
students  will  be  better  prepared  for  future  careers  in  a 
public  or  government  community  development  agency  which  may 
require  them  to  evaluate  the  effects  of  the  programs  they  are 
responsible  for  implementing.  The  skills  of  evaluation 
research  will  enable  future  community  development  practitioners 
to  measure  the  effects  of  related  programs  and  hence,  to  meet 
the  demands  of  accountability  from  funders  as  clearly,  when 
the  monies  become  scarce  they  are  allocated  only  to  those  who 
can  best  demonstrate  advantageous  results. 

3.  It  would  be  advisable  for  graduate  students  in 
community  development  to  hold  a  voluntary  association  or 
practicum  with  a  community  development  agency  such  as  the 
Edmonton  Social  Planning  Council.  A  practicum  would  provide 
students  with  an  opportunity  to  utilize  theoretical  skills 
in  a  practical  situation,  such  as  assisting  in  the  writing 
of  evaluation  reports. 

4.  The  development  of  more  specialized  evaluation 
organizations  is  encouraged  (e.g.  Ontario  Institute  of  Studies 
in  Education)  and  particularly  one  for  the  field  of  community 
development.  In  this  way,  a  central  resource  center  will  be 
created  towards  which  the  community  can  look  for  reference 
material,  consultation  and  possible  contract  work.  It  appears 
that  the  Edmonton  Social  Planning  Council  is  making  a 
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significant  effort  towards  developing  such  a  center. 

5.  The  results  of  evaluation  sutdies  should  be 
disseminated  more  effectively  to  and  beyond  the  community  level. 
The  benefits  from  sharing  ideas  and  problems  related  to  evalu¬ 
ation  studied  can  lead  to  more  comparative  studies  which  will 
provide  useful  information  in  adding  to  the  knowledge  base 

and  providing  more  useful  guidelines  for  funding  agents. 

6.  Finally,  it  is  suggested  that  community  develop¬ 
ment  researchers  ensure  that  evaluation  results  are  dissemin¬ 
ated  to  the  people  of  a  community  in  a  language  and  form  that 
they  can  easily  comprehend. 
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Introduction 


The  purpose  of  this  appendix  is  to  describe  how  the 
framework  developed  in  this  thesis  might  be  applied  to  a 
case  study.  It  is  not  a  detailed  application  but  a  general 
discussion  of  how  the  quantitative  and  qualitative  measures 
within  the  systems-process-context  approach  could  be  applied 
to  a  real  program. 

The  case  study  of  The  Volunteer  Training  Program  has 
been  selected  from  amongst  the  various  community  development 
programs  that  are  implemented  and  supported  by  the  Edmonton 
Social  Planning  Council.  The  following  discussion  will  briefly 
describe  The  Volunteer  Training  Program  and  then  apply  most 
of  those  questions  which  were  listed  under  items  to  be  con¬ 
sidered  in  Table  2  (Chapter  4)  to  this  particular  case. 

Program  Description 

The  Volunteer  Training  Program  is  directed  at  a  core 
group  of  people  from  voluntary  organizations  and  community 
groups  in  Edmonton  who  have  expressed  an  interest  to  the 
Social  Planning  Council  in  obtaining  additional  skills  in  small 
group  leadership,  research  and  knowledge  of  locations  and  use 
of  community  resources.  The  program  provides  its  support  and 
training  to  volunteer  groups  through  a  series  of  workshops 
which  are  organized  by  staff  of  the  Council.  The  workshops 
incorporate  such  items  as  films,  presentations  by  speakers, 
and  informal  discussions  between  program  staff  and  recipients 
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which  center  around  such  issues  as:  how  to  mobilize  and 
organize  a  community;  how  to  tap  local  and  city-wide 
resources;  how  to  discover  community  needs  and  how  to  plan 
the  best  ways  to  meet  these;  how  to  understand  human  group 
dynamics;  how  to  understand  systems  (e.g.  social,  economic, 
political,  cultural,  physical);  and  finally,  how  to  analyze 
policy.  The  workshops  are  intended  to  help  people  to  discover 
more  effective  and  satisfying  ways  of  doing  their  tasks, 
thereby  ultimately  improving  their  programs  for  their  clients. 

An  Application  of  The  Systems-Process-Context 

Approach  to  The  Volunteer  Training  Program 

In  discussing  the  application  of  the  systems-process- 
context  approach  to  The  Volunteer  Training  Program,  reference 
will  be  made  to  the  questions  listed  under  items  to  be  con¬ 
sidered  in  Table  2  (pp.  144-150) .  In  this  way  an  illustration 
will  be  given  of  how  the  framework  developed  in  this  thesis 
could  be  utilized  to  evaluate  a  community  development  project. 

We  begin  by  considering  those  items/questions  which 
measure  the  effects  of  a  service .  It  appears  that  the  program 
reflects  a  true  community  development  orientation  (1,2, a).1 

The  types  and  numbers  of  recipients  from  the  service  (1,2,6) 
can  be  easily  identified  upon  examination  of  those  people 
making  up  the  core  group  of  interested  volunteers.  Also,  it 
appears  that  the  service  is  made  accessible  to  the  entire 

■’’Throughout  this  appendix  the  number  and  letters  in 
parentheses  refer  to  the  columns  in  Table  2  of  this  thesis. 
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community  through  flyers  and  pamphlets  circulated  in  the 
communities  by  the  Edmonton  Social  Planning  Council  (E.S.P.C.). 
The  visible  effects  of  the  service  on  recipients  (1,2,6)  can 
be  determined  by  examining  the  degree  to  which  their  partici¬ 
pation  and  involvement  in  other  related  learning  experiences 
within  the  community  changes.  The  service  appears  to  have  a 
sound  basis  (l,2,d),  since,  financially,  it  is  supported  by 
the  E.S.P.C.  which  receives  its  monies  from  the  United  Way 
and  City  grants.  In  addition,  community  support  is  continually 
supplemented  by  those  interested  in  volunteer  groups  behind 
the  program.  A  self-evaluation  process  (l,2,e)  is  undertaken 
by  the  program  recipients  who  are  asked  to  describe  to  the 
E.S.P.C.  staff  how  they  perceived  changes  in  their  community 
involvement  as  a  result  of  having  taken  part  in  the  program. 
Expansion  of  the  service  warranted  in  terms  of  population  in¬ 
crease  or  decrease  (l,2,f)  is  not  a  directly  relevant  measure 
as  the  expansion  of  the  program  is  usually  dependent  on  the 
degree  of  interest  expressed  by  people  in  need  of  the  service, 
regardless  of  population  size. 

In  the  case  of  The  Volunteer  Training  Program,  most 
questions  listed  under  items  to  be  considered  related  to 
measuring  facilities  are  not  applicable  as  it  is  not  a  facility 
orient  program.  Thus,  reference  to  where  the  program  workshops 
are  held  (e.g.  Grant  McEwan  college)  and  the  frequency  of 
their  use  can  be  employed  to  measure  whether  there  are  a 
sufficient  number  of  facilities  to  accommodate  the  program 
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service  in  the  community  (2,2, a). 

The  third  set  of  items  to  be  considered  relates  to 
changes  in  administrative  decision-making  processes.  In  the 
case  of  The  Volunteer  Training  Program,  it  appears  that 
relatively  formal  techniques  are  used  to  collect  information 
prior  to  the  implementation  of  a  program  (1,2, a)  such  as 
telephone-inquiries.  The  information  collected  appears  to 
be  clear  (l,2,b)  as  the  questions  posed  (e.g.  "Do  you  belong 
to  a  group  that  would  like  this  training?",  or  "Would  you 
like  to  be  part  of  our  training  team?")  are  easily  understood 
and  answered  by  volunteers.  Responses  to  the  information 
(l,2,c)  are  representative  of  input  from  special  groups  in 
the  community.  An  evaluation  process  does  occur  (l,2,d)  the 
results  of  which  appears  to  aid  the  E.S.P.C.  staff  in  distill¬ 
ing  the  numbers  of  alternatives  reflecting  the  priorities  identif¬ 
ied  by  the  community  groups.  It  is  these  priority  areas  which 
are  discussed  in  the  workshops.  Collaborative  efforts  be¬ 
tween  specialist  and  community  groups  (l,2,e)  appear  to  occur 
as  reflected  in  the  Board-Staff  Workshops  whereby  both  staff 
and  volunteers  work  out  establishing  the  areas  of  felt  needs. 
The  nature  of  contact  between  client  groups  and  administrators 
(l,2,f)  is  primarily  in  terms  of  informal  discussions  which 
facilitate  input  from  both  sides  into  the  administrative 
decision-making  process. 

The  next  concern  is  with  the  degree  of  enhancement 
in  the  individual's  personal  condition.  In  the  case  of  The 
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Volunteer  Training  Program,  it  is  not  difficult  to  identify 
the  types  of  people  involved  in  the  program  in  terms  of 
their  numbers,  age,  class,  socio-economic  status,  marital 
status,  and  cultural  background  (2, 2, a).  It  appears  that 
behavioural  changes  are  measured  by  the  E.S.P.C.  staff  who 
ask  the  volunteers  to  describe  how  they  perceive  changes  in 
their  behaviour  as  a  result  of  having  taken  part  in  the  pro¬ 
gram.  The  following  list  of  quotations  and  reactions  cover 
a  wide  variety  of  behavioural  changes  (e.g.  in  knowledge, 
skills,  aspirations  and  attitudes)  which  could  indicate  the 
success  of  the  program  in  creating  an  impact  within  the 
community.  This  could  also  suggest  directions  for  the  future 
improvement  of  the  program. 

"I  will  be  less  frustrated  because  I  have  learned 
how  to  show  change  is." 

"I  have  been  able  to  hold  better  meetings,  get  more 
data  and  had  better  participation  by  those  attending." 

"Volunteers  are  made  to  feel  important  and  this  gives 
me  more  confidence  to  tackle  other  commitments." 

"Time  is  spent  in  a  nice  effective  manner,  that's 
very  satisfying  to  me." 

"I  appreciate  being  able  to  understand  a  philosophy 
and  put  it  into  action  in  practical  terms  (e.g. 
planning  a  conference) . " 

"I  must  be  seen  to  be  more  aware  and  competent  because 
I  have  been  asked  to  accept  a  great  leadership  role 
in  my  community." 

"I  am  better  prepared  to  work  with  other  volunteers 
successfully . " 
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"I  am  not  doing  more  but  have  a  better  understanding 

of  what  I  am  doing  and  why. "2 

The  measurement  of  changes  reflecting  an  improving 
sense  of  community  identity  is  the  fifth  area  of  concern. 
Examples  of  self-help  schemes  (3, 2, a)  such  as  rape  crisis 
centres ,  and  day  care  centres ,  exist  in  the  community  as  a 
result  of  trainees  from  the  program  having  made  use  of  local 
community  resources.  The  core  group  of  volunteers  partici¬ 
pating  in  the  program  are  those  local  leaders  who  are  employed 
to  a  large  degree  in  mobilizing  the  community  towards  action 
(3,2, b).  Moreover,  they  appear  to  have  been  effective  in 
their  efforts  (3,2,c)  as  more  people  have  expressed  an  inter¬ 
est  to  the  E.S.P.C.  in  becoming  involved  in  the  program. 

It  appears  that  the  program  has  prompted  citizens  to  become 
creative  in  spreading  ideas  and  exploring  new  areas  of  develop¬ 
ment  (3,2,d)  as  exemplified  in  the  setting  up  of  a  community 
theatre  in  Montrose.  A  mutual  education  process  is  provided 
in  the  program  workshops  which  disseminate  information,  relat¬ 
ing  to  such  matters  as  community  planning  techniques,  that  are 
relevant  to  creating  within  the  program  recipients  a  better 
understanding  of  current  community  issues  and  the  ways  to 
meet  needs  (3,2,d).  The  program  life  has  increased  since  it 
began  in  1977  providing  an  opportunity  for  more  citizens 
to  participate  in  it  (3,2,c). 

2Edmonton  Social  Planning  Council,  Proposal  Develop¬ 
ment  Regarding  Community  Development  Corporation  Implementation 
in  Edmonton,  (Edmonton)  pi  8"7~ 
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The  next  area  of  concern  is  with  the  measurement  of 
improvements  (if  any)  in  local  decision-making  processes.  It 
is  the  volunteers  in  the  core  groups  who  dominate  the  decision¬ 
making  (4, 2, a)  which  are  essentially  of  an  open  kind,  reflect¬ 
ing  collaborative  efforts  and  consensus  (4,2,b).  The  decision¬ 
making  base  appears  to  be  quite  broad  (4,2,0)  as  it  is  open 
to  any  number  of  the  various  types  of  interested  citizens. 

Input  into  the  decision-making  processes  is  partially  obtained 
from  the  E.S.P.C.  staff  (4,2,d)  who  encourage  and  train  the 
volunteer  to  understand  the  effects  of  their  decisions  in  the 
larger  environments  (e.g.  government) .  Citizen  commissions 
and  community  task  forces  are  those  structures  (4,2,e)  which 
are  used  in  the  decision-making  processes  revolving  around 
major  policy  issues  in  the  community. 

The  ability  to  maintain  a  viable  community  in  relation 
to  the  larger  environment  is  the  penultimate  area  of  concern. 

In  the  case  of  The  Volunteer  Training  Program,  it  appears 
that  it  has  been  effective  in  creating  social/political 
contacts  between  communities  and  agencies/groups  in  the  larger 
contexts  (5, 2, a).  For  example,  trainees  of  the  program  have 
been  asked  to  assist  other  communities  in  setting  up  a  group 
of  seminars  in  community  planning  and  have  led  volunteer 
groups  from  all  over  Edmonton  in  the  Mayor's  Conference  on 
Neighbourhood  Planning.  It  appears  that  the  program  has 
generated  support  from  sources  other  than  government  (5,2,b) 
and  has  been  effective  in  attracting  expertise  from  outside 
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the  E.S.P.C.  (5,2,0).  For  example  personnel  from  the  Pro¬ 
vincial  Department  of  Preventive,  Social  Services,  have 
examined  and  used  the  evaluation  methods  employed  in  The 
Volunteer  Training  Programs.  The  degree  of  dependency  of 
the  program  on  the  larger  environment  by  the  E.S.P.C.  staff 
(5,2,d)  appears  to  be  lessening.  That  is,  because  of  a 
shortage  of  staff  to  implement  Volunteer  Training  Programs, 
the  E.S.P.C.  is  sending  "packages"  to  the  communities.  These 
"packages"  include  written  material  on  what  seminars  and 
workshops  should  cover.  It  then  becomes  the  responsibility 
of  the  community  groups  to  implement  the  Training  Programs 
themselves  based  on  the  information  supplied  in  the  "packages". 

The  final  concern  is  with  increased  government  and/or 
political  commitment.  In  the  case  of  the  program,  it  appears 
that  financial  support  from  governmental  agencies  has  been 
and  is  being  acquired  (6, 2, a)  as  well  as  political  support 
from  local  groups  (6,2,b) .  There  is  also  political  commitment 
from  outside  the  community  (6,2,c).  The  Edmonton  Social 
Planning  Council  recognizes  that  if  voluntary  efforts  are  to 
become  effective  at  implementing  change,  they  must  meet  the 
demands  of  the  larger  societal  milieu.  In  order  to  achieve 
this,  an  underlying  philosophy  behind  The  Volunteer  Training 
Program  is  that  trainees  should  be  encouraged  to  make  contacts 
with  local  politicians,  bureaucrats,  civil  servants  and 
other  local  elite  groups  all  of  whose  political  and  financial 
support  is  beneficial  to  the  continuance  of  the  programs  that 
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volunteers  initiate. 

In  summary  and  conclusion,  it  appears  that, prima 
facie  at  least,  those  quantitative  and  qualitative  measures 
outlined  in  the  systems-process-context  approach  can  be 
applied  to  evaluating  the  effects  of  change  initiated  by  a 
real  program. 


